2024-08-19T01:18:38.529 INFO:root:teuthology version: 1.2.0 2024-08-19T01:18:38.556 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T01:18:38.661 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7860119' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 78215f5a1cf5eaf321d506a66539942e12851f34 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1664 sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 sleep_before_teardown: 0 subset: 36/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 78215f5a1cf5eaf321d506a66539942e12851f34 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIqewOEyH9c/i04u3XowU6uvb5oeR4cy2E4KBPRuGwp4bbuf9vuBBgD5HbTpdoKx1VegHI9Fre8sFCvivf0aJ14= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJO5qq5IJnYq3Zi5yz1txdg80OjkW2fF/AB2ack7sw2x1Pl12sDACk4xuEbmCIFd8+BB6Fj+lQbffrHoDGoJtgM= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJhKzgmFE+3OW1Ys10fZLXPrFcu35FmbU8S31pfZtUYzTajrDHnrbAidjHIFXdW5UyARU0ZB0weGPajk3skKE0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-18_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-19T01:18:38.662 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa; will attempt to use it 2024-08-19T01:18:38.662 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks 2024-08-19T01:18:38.662 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-19T01:18:38.664 INFO:teuthology.task.internal:Checking packages... 2024-08-19T01:18:38.686 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'de09dd018de4baddfa6ca4d43bc8fb784cf75833' 2024-08-19T01:18:38.686 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T01:18:38.686 INFO:teuthology.packaging:ref: None 2024-08-19T01:18:38.686 INFO:teuthology.packaging:tag: None 2024-08-19T01:18:38.686 INFO:teuthology.packaging:branch: squid 2024-08-19T01:18:38.686 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:18:38.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-19T01:18:38.882 INFO:teuthology.task.internal:Found packages for ceph version 19.1.0-1386.g78215f5a 2024-08-19T01:18:38.883 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-19T01:18:38.890 INFO:teuthology.task.internal:no buildpackages task found 2024-08-19T01:18:38.891 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-19T01:18:38.899 INFO:teuthology.task.internal:Saving configuration 2024-08-19T01:18:38.919 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-19T01:18:38.926 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-19T01:18:38.952 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 01:13:29.202876', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIqewOEyH9c/i04u3XowU6uvb5oeR4cy2E4KBPRuGwp4bbuf9vuBBgD5HbTpdoKx1VegHI9Fre8sFCvivf0aJ14='} 2024-08-19T01:18:38.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 01:13:29.199430', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJO5qq5IJnYq3Zi5yz1txdg80OjkW2fF/AB2ack7sw2x1Pl12sDACk4xuEbmCIFd8+BB6Fj+lQbffrHoDGoJtgM='} 2024-08-19T01:18:38.993 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 01:13:29.201269', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJhKzgmFE+3OW1Ys10fZLXPrFcu35FmbU8S31pfZtUYzTajrDHnrbAidjHIFXdW5UyARU0ZB0weGPajk3skKE0='} 2024-08-19T01:18:38.993 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-19T01:18:39.001 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-08-19T01:18:39.001 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-08-19T01:18:39.001 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-08-19T01:18:39.001 INFO:teuthology.run_tasks:Running task console_log... 2024-08-19T01:18:39.083 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f65fa393be0>, signals=[15]) 2024-08-19T01:18:39.083 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-19T01:18:39.090 INFO:teuthology.task.internal:Opening connections... 2024-08-19T01:18:39.090 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:18:39.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T01:18:39.183 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-08-19T01:18:39.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T01:18:39.256 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-08-19T01:18:39.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T01:18:39.330 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-19T01:18:39.338 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-08-19T01:18:39.356 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-08-19T01:18:39.357 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-08-19T01:18:39.415 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-08-19T01:18:39.415 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-08-19T01:18:39.415 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-08-19T01:18:39.415 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-08-19T01:18:39.415 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-08-19T01:18:39.415 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-08-19T01:18:39.415 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T01:18:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-08-19T01:18:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-08-19T01:18:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T01:18:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-08-19T01:18:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T01:18:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T01:18:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T01:18:39.417 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-08-19T01:18:39.441 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-08-19T01:18:39.459 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-08-19T01:18:39.459 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-08-19T01:18:39.517 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="9" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="9" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el9" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:LOGO="fedora-logo-icon" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T01:18:39.518 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T01:18:39.519 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-08-19T01:18:39.541 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-08-19T01:18:39.559 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-08-19T01:18:39.559 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-08-19T01:18:39.617 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-08-19T01:18:39.617 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-08-19T01:18:39.617 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T01:18:39.618 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T01:18:39.619 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-08-19T01:18:39.640 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-19T01:18:39.651 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-19T01:18:39.664 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-19T01:18:39.665 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-08-19T01:18:39.667 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-08-19T01:18:39.669 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-08-19T01:18:39.686 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-19T01:18:39.694 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-19T01:18:39.694 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-08-19T01:18:39.724 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-08-19T01:18:39.727 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-08-19T01:18:39.786 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-19T01:18:39.853 INFO:teuthology.run_tasks:Running task kernel... 2024-08-19T01:18:39.867 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-08-19T01:18:39.868 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-08-19T01:18:39.868 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-08-19T01:18:39.868 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T01:18:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T01:18:39.885 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-08-19T01:18:39.944 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-480.el9.x86_64 2024-08-19T01:18:39.944 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-480.el9.x86_64 2024-08-19T01:18:39.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-08-19T01:18:42.267 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-08-19T01:18:44.317 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-08-19T01:18:50.877 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.6 MB 00:02 2024-08-19T01:18:52.436 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 18 kB 00:00 2024-08-19T01:18:53.817 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 18 MB/s | 22 MB 00:01 2024-08-19T01:18:59.621 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-19T01:19:01.560 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T01:19:01.561 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T01:19:01.658 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-08-19T01:19:01.659 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-08-19T01:19:01.660 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-08-19T01:19:01.757 DEBUG:teuthology.orchestra.run.smithi064:> echo no | sudo yum reinstall kernel || true 2024-08-19T01:19:02.292 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 01:18:59 AM UTC. 2024-08-19T01:19:02.437 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-08-19T01:19:02.446 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-08-19T01:19:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-08-19T01:19:02.446 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-08-19T01:19:02.446 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-08-19T01:19:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T01:19:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:19:02.447 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-08-19T01:19:02.447 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-08-19T01:19:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:19:02.447 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.5 M 2024-08-19T01:19:02.447 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-08-19T01:19:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:Operation aborted. 2024-08-19T01:19:02.505 INFO:teuthology.orchestra.run.smithi064.stdout:Is this ok [y/N]: 2024-08-19T01:19:02.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum reinstall -y kernel || true 2024-08-19T01:19:03.033 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:04 ago on Mon 19 Aug 2024 01:18:59 AM UTC. 2024-08-19T01:19:03.162 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:19:03.163 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.5 M 2024-08-19T01:19:03.164 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-08-19T01:19:03.164 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-08-19T01:19:03.748 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-480.el9.x86_64.rpm 3.1 MB/s | 1.5 MB 00:00 2024-08-19T01:19:03.748 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-08-19T01:19:03.748 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.6 MB/s | 1.5 MB 00:00 2024-08-19T01:19:03.784 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-08-19T01:19:04.033 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-08-19T01:19:04.033 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-08-19T01:19:04.058 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-08-19T01:19:04.059 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-08-19T01:19:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-08-19T01:19:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T01:19:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T01:19:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T01:19:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T01:19:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T01:19:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:19:05.883 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalled: 2024-08-19T01:19:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T01:19:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:19:05.884 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-08-19T01:19:06.069 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T01:19:06.125 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T01:19:06.125 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T01:19:06.125 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T01:19:06.125 DEBUG:teuthology.parallel:result is None 2024-08-19T01:19:06.125 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-19T01:19:06.133 INFO:teuthology.task.internal:Creating test directory... 2024-08-19T01:19:06.133 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T01:19:06.135 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T01:19:06.138 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T01:19:06.156 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-19T01:19:06.164 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-19T01:19:06.173 INFO:teuthology.task.internal:Creating archive directory... 2024-08-19T01:19:06.174 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T01:19:06.190 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T01:19:06.196 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T01:19:06.318 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-19T01:19:06.325 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-19T01:19:06.325 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T01:19:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T01:19:06.343 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T01:19:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T01:19:06.360 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T01:19:06.362 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T01:19:06.385 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T01:19:06.402 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T01:19:06.419 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T01:19:06.432 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T01:19:06.449 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T01:19:06.456 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T01:19:06.475 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T01:19:06.477 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-19T01:19:06.483 INFO:teuthology.task.internal:Configuring sudo... 2024-08-19T01:19:06.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T01:19:06.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T01:19:06.488 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T01:19:06.516 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-19T01:19:06.525 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-19T01:19:06.525 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T01:19:06.551 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T01:19:06.556 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T01:19:06.575 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T01:19:06.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T01:19:06.696 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T01:19:06.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T01:19:06.817 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:19:06.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T01:19:06.882 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T01:19:06.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T01:19:06.977 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T01:19:07.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T01:19:07.111 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:19:07.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T01:19:07.178 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T01:19:07.224 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T01:19:07.286 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T01:19:07.329 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T01:19:07.374 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:19:07.374 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T01:19:07.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-08-19T01:19:07.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-08-19T01:19:07.446 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-08-19T01:19:07.480 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T01:19:07.486 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T01:19:07.521 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T01:19:07.953 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-19T01:19:07.960 INFO:teuthology.task.internal:Starting timer... 2024-08-19T01:19:07.960 INFO:teuthology.run_tasks:Running task pcp... 2024-08-19T01:19:08.017 INFO:teuthology.run_tasks:Running task selinux... 2024-08-19T01:19:08.070 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-08-19T01:19:08.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-08-19T01:19:08.120 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-08-19T01:19:08.122 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-08-19T01:19:08.180 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-08-19T01:19:08.181 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-08-19T01:19:08.230 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-08-19T01:19:08.232 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-19T01:19:08.232 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-08-19T01:19:08.257 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-08-19T01:19:08.258 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-08-19T01:19:08.284 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-08-19T01:19:08.285 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-08-19T01:19:08.307 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-08-19T01:19:08.308 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2024-08-19T01:19:08.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T01:19:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T01:19:08.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T01:19:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T01:19:08.362 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T01:19:08.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T01:19:08.389 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-19T01:19:08.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-08-19T01:19:08.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-08-19T01:19:08.441 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-08-19T01:19:08.466 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-19T01:19:08.476 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-19T01:19:08.487 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-19T01:19:08.488 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi123.front.sepia.ceph.com,smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-19T01:21:11.863 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2024-08-19T01:21:11.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-08-19T01:21:11.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T01:21:11.940 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-08-19T01:21:12.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-08-19T01:21:12.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-08-19T01:21:12.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T01:21:12.095 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-08-19T01:21:12.174 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-08-19T01:21:12.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-08-19T01:21:12.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T01:21:12.249 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-08-19T01:21:12.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-08-19T01:21:12.330 INFO:teuthology.run_tasks:Running task clock... 2024-08-19T01:21:12.341 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-19T01:21:12.341 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T01:21:12.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T01:21:12.344 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T01:21:12.344 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T01:21:12.346 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T01:21:12.346 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T01:21:12.374 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T01:21:12.376 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T01:21:12.390 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T01:21:12.391 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T01:21:12.400 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T01:21:12.415 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T01:21:12.416 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-08-19T01:21:12.416 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-08-19T01:21:12.428 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-08-19T01:21:12.428 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-08-19T01:21:12.440 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-08-19T01:21:12.442 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T01:21:12.442 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T01:21:12.451 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-08-19T01:21:12.456 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T01:21:12.457 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T01:21:12.465 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T01:21:12.480 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T01:21:12.511 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-08-19T01:21:12.512 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-08-19T01:21:12.515 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T01:21:12.515 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-08-19T01:21:12.515 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.515 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.515 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.515 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.516 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T01:21:12.516 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-08-19T01:21:12.516 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.516 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.516 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.516 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.537 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-08-19T01:21:12.540 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T01:21:12.541 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-08-19T01:21:12.541 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.541 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.541 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.541 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T01:21:12.541 INFO:teuthology.run_tasks:Running task install... 2024-08-19T01:21:12.549 DEBUG:teuthology.task.install:project ceph 2024-08-19T01:21:12.549 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'}} 2024-08-19T01:21:12.549 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'} 2024-08-19T01:21:12.550 INFO:teuthology.task.install:Using flavor: default 2024-08-19T01:21:12.555 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-19T01:21:12.555 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-08-19T01:21:12.555 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'tag': None, 'wait_for_package': False} 2024-08-19T01:21:12.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:21:12.557 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'tag': None, 'wait_for_package': False} 2024-08-19T01:21:12.557 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:21:12.558 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'tag': None, 'wait_for_package': False} 2024-08-19T01:21:12.559 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:21:12.706 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/ 2024-08-19T01:21:12.707 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1382.gde09dd01 2024-08-19T01:21:12.710 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/ 2024-08-19T01:21:12.710 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1382.gde09dd01 2024-08-19T01:21:12.720 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/ 2024-08-19T01:21:12.720 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1382.gde09dd01 2024-08-19T01:21:12.816 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T01:21:12.816 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:21:12.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T01:21:12.825 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T01:21:12.825 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:21:12.825 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T01:21:12.827 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T01:21:12.827 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:21:12.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T01:21:12.842 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-19T01:21:12.842 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/de09dd018de4baddfa6ca4d43bc8fb784cf75833/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T01:21:12.853 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-19T01:21:12.853 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/de09dd018de4baddfa6ca4d43bc8fb784cf75833/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T01:21:12.854 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-19T01:21:12.854 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/de09dd018de4baddfa6ca4d43bc8fb784cf75833/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T01:21:12.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T01:21:12.919 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T01:21:12.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T01:21:12.982 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T01:21:12.994 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T01:21:12.995 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T01:21:13.047 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-08-19T01:21:13.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-08-19T01:21:13.057 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-08-19T01:21:13.059 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-08-19T01:21:13.060 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-08-19T01:21:13.060 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-08-19T01:21:13.280 INFO:teuthology.orchestra.run.smithi064.stdout:45 files removed 2024-08-19T01:21:13.284 INFO:teuthology.orchestra.run.smithi123.stdout:45 files removed 2024-08-19T01:21:13.285 INFO:teuthology.orchestra.run.smithi161.stdout:45 files removed 2024-08-19T01:21:13.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-19T01:21:13.311 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-19T01:21:13.313 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-19T01:21:13.850 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 317 kB/s | 84 kB 00:00 2024-08-19T01:21:13.856 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 311 kB/s | 84 kB 00:00 2024-08-19T01:21:13.879 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 283 kB/s | 84 kB 00:00 2024-08-19T01:21:14.149 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-08-19T01:21:14.163 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-08-19T01:21:14.234 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-08-19T01:21:14.424 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-08-19T01:21:14.439 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-08-19T01:21:14.510 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-08-19T01:21:15.227 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-19T01:21:15.291 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.2 MB 00:00 2024-08-19T01:21:16.382 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-08-19T01:21:17.125 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-08-19T01:21:18.353 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-08-19T01:21:21.506 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 20 MB 00:04 2024-08-19T01:21:22.359 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.6 MB 00:01 2024-08-19T01:21:22.945 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-19T01:21:23.827 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 18 kB 00:00 2024-08-19T01:21:24.581 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 35 MB/s | 22 MB 00:00 2024-08-19T01:21:25.204 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 18 kB 00:01 2024-08-19T01:21:25.991 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 34 MB/s | 22 MB 00:00 2024-08-19T01:21:26.398 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.6 MB 00:01 2024-08-19T01:21:28.007 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 18 kB 00:00 2024-08-19T01:21:29.086 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 23 MB/s | 22 MB 00:00 2024-08-19T01:21:29.920 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-19T01:21:31.514 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-19T01:21:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-19T01:21:32.125 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-19T01:21:32.138 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.48.2-2.el9.x86_64 is already installed. 2024-08-19T01:21:32.139 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-19T01:21:32.140 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-19T01:21:32.141 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-19T01:21:32.143 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-19T01:21:32.146 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-19T01:21:32.146 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-6.4.0-3.el9.x86_64 is already installed. 2024-08-19T01:21:32.207 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-08-19T01:21:32.223 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-08-19T01:21:32.223 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-08-19T01:21:32.223 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-08-19T01:21:32.223 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-08-19T01:21:32.223 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-19T01:21:32.223 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 6.6 k 2024-08-19T01:21:32.223 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 896 k 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 145 k 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 1.5 M 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 146 k 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 3.6 M 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 7.4 M 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 49 k 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 11 M 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 49 M 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 285 k 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 766 k 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 225 k 2024-08-19T01:21:32.224 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 27 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 18 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 32 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 753 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 127 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-08-19T01:21:32.225 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 188 k 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 369 k 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 350 k 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 117 k 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 85 k 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.1 M 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 170 k 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.4 M 2024-08-19T01:21:32.226 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.2 M 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 21 M 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 32 k 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 2.4 M 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 249 k 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 4.7 M 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 17 M 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 16 k 2024-08-19T01:21:32.227 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 25 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 163 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T01:21:32.228 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 507 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T01:21:32.229 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 45 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 136 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T01:21:32.230 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T01:21:32.231 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T01:21:32.232 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T01:21:32.233 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T01:21:32.234 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout:Install 148 Packages 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 2 Packages 2024-08-19T01:21:32.235 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:21:32.236 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 218 M 2024-08-19T01:21:32.236 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-08-19T01:21:32.613 INFO:teuthology.orchestra.run.smithi161.stdout:(1/150): ceph-19.1.0-1382.gde09dd01.el9.x86_64. 50 kB/s | 6.6 kB 00:00 2024-08-19T01:21:32.821 INFO:teuthology.orchestra.run.smithi161.stdout:(2/150): ceph-fuse-19.1.0-1382.gde09dd01.el9.x8 4.2 MB/s | 896 kB 00:00 2024-08-19T01:21:32.947 INFO:teuthology.orchestra.run.smithi161.stdout:(3/150): ceph-base-19.1.0-1382.gde09dd01.el9.x8 12 MB/s | 5.4 MB 00:00 2024-08-19T01:21:32.972 INFO:teuthology.orchestra.run.smithi161.stdout:(4/150): ceph-immutable-object-cache-19.1.0-138 966 kB/s | 145 kB 00:00 2024-08-19T01:21:33.097 INFO:teuthology.orchestra.run.smithi161.stdout:(5/150): ceph-mds-19.1.0-1382.gde09dd01.el9.x86 16 MB/s | 2.4 MB 00:00 2024-08-19T01:21:33.164 INFO:teuthology.orchestra.run.smithi161.stdout:(6/150): ceph-mgr-19.1.0-1382.gde09dd01.el9.x86 7.9 MB/s | 1.5 MB 00:00 2024-08-19T01:21:33.331 INFO:teuthology.orchestra.run.smithi161.stdout:(7/150): ceph-mon-19.1.0-1382.gde09dd01.el9.x86 20 MB/s | 4.7 MB 00:00 2024-08-19T01:21:33.706 INFO:teuthology.orchestra.run.smithi161.stdout:(8/150): ceph-common-19.1.0-1382.gde09dd01.el9. 17 MB/s | 21 MB 00:01 2024-08-19T01:21:33.748 INFO:teuthology.orchestra.run.smithi161.stdout:(9/150): ceph-selinux-19.1.0-1382.gde09dd01.el9 608 kB/s | 25 kB 00:00 2024-08-19T01:21:33.783 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-19T01:21:33.784 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-19T01:21:33.796 INFO:teuthology.orchestra.run.smithi064.stdout:Package NetworkManager-1:1.48.2-2.el9.x86_64 is already installed. 2024-08-19T01:21:33.797 INFO:teuthology.orchestra.run.smithi064.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-19T01:21:33.798 INFO:teuthology.orchestra.run.smithi064.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-19T01:21:33.798 INFO:teuthology.orchestra.run.smithi064.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-19T01:21:33.800 INFO:teuthology.orchestra.run.smithi064.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-19T01:21:33.803 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-19T01:21:33.803 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsprogs-6.4.0-3.el9.x86_64 is already installed. 2024-08-19T01:21:33.868 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-08-19T01:21:33.877 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-08-19T01:21:33.877 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-08-19T01:21:33.877 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-08-19T01:21:33.877 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-08-19T01:21:33.877 INFO:teuthology.orchestra.run.smithi064.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-19T01:21:33.877 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 6.6 k 2024-08-19T01:21:33.877 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 896 k 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 145 k 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 1.5 M 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 146 k 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 3.6 M 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 7.4 M 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 49 k 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 11 M 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 49 M 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 285 k 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 766 k 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 225 k 2024-08-19T01:21:33.878 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 27 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 18 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 32 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 753 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 127 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-08-19T01:21:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 188 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 369 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 350 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 117 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 85 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.1 M 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 170 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.4 M 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.2 M 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T01:21:33.880 INFO:teuthology.orchestra.run.smithi064.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 21 M 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 32 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 2.4 M 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 249 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 4.7 M 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 17 M 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 16 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 25 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T01:21:33.881 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 163 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 507 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T01:21:33.882 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T01:21:33.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 45 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 136 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T01:21:33.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T01:21:33.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T01:21:33.886 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T01:21:33.887 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-19T01:21:33.888 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout:Install 148 Packages 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-08-19T01:21:33.889 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:21:33.890 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 218 M 2024-08-19T01:21:33.890 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-08-19T01:21:34.040 INFO:teuthology.orchestra.run.smithi161.stdout:(10/150): ceph-radosgw-19.1.0-1382.gde09dd01.el 15 MB/s | 11 MB 00:00 2024-08-19T01:21:34.090 INFO:teuthology.orchestra.run.smithi161.stdout:(11/150): cephfs-mirror-19.1.0-1382.gde09dd01.e 4.4 MB/s | 225 kB 00:00 2024-08-19T01:21:34.157 INFO:teuthology.orchestra.run.smithi161.stdout:(12/150): cephfs-shell-19.1.0-1382.gde09dd01.el 407 kB/s | 27 kB 00:00 2024-08-19T01:21:34.224 INFO:teuthology.orchestra.run.smithi161.stdout:(13/150): libcephfs-devel-19.1.0-1382.gde09dd01 476 kB/s | 32 kB 00:00 2024-08-19T01:21:34.324 INFO:teuthology.orchestra.run.smithi161.stdout:(14/150): libcephfs2-19.1.0-1382.gde09dd01.el9. 7.4 MB/s | 753 kB 00:00 2024-08-19T01:21:34.391 INFO:teuthology.orchestra.run.smithi161.stdout:(15/150): libcephsqlite-19.1.0-1382.gde09dd01.e 2.4 MB/s | 163 kB 00:00 2024-08-19T01:21:34.460 INFO:teuthology.orchestra.run.smithi161.stdout:(16/150): librados-devel-19.1.0-1382.gde09dd01. 1.8 MB/s | 127 kB 00:00 2024-08-19T01:21:34.499 INFO:teuthology.orchestra.run.smithi064.stdout:(1/150): ceph-19.1.0-1382.gde09dd01.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-08-19T01:21:34.500 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-08-19T01:21:34.627 INFO:teuthology.orchestra.run.smithi161.stdout:(17/150): ceph-osd-19.1.0-1382.gde09dd01.el9.x8 12 MB/s | 17 MB 00:01 2024-08-19T01:21:34.669 INFO:teuthology.orchestra.run.smithi161.stdout:(18/150): libradosstriper1-19.1.0-1382.gde09dd0 2.4 MB/s | 507 kB 00:00 2024-08-19T01:21:34.732 INFO:teuthology.orchestra.run.smithi064.stdout:(2/150): ceph-fuse-19.1.0-1382.gde09dd01.el9.x8 3.8 MB/s | 896 kB 00:00 2024-08-19T01:21:34.761 INFO:teuthology.orchestra.run.smithi161.stdout:(19/150): python3-ceph-argparse-19.1.0-1382.gde 492 kB/s | 45 kB 00:00 2024-08-19T01:21:34.849 INFO:teuthology.orchestra.run.smithi064.stdout:(3/150): ceph-immutable-object-cache-19.1.0-138 1.2 MB/s | 145 kB 00:00 2024-08-19T01:21:35.150 INFO:teuthology.orchestra.run.smithi064.stdout:(4/150): ceph-base-19.1.0-1382.gde09dd01.el9.x8 6.6 MB/s | 5.4 MB 00:00 2024-08-19T01:21:35.208 INFO:teuthology.orchestra.run.smithi064.stdout:(5/150): ceph-mds-19.1.0-1382.gde09dd01.el9.x86 6.7 MB/s | 2.4 MB 00:00 2024-08-19T01:21:35.261 INFO:teuthology.orchestra.run.smithi161.stdout:(20/150): python3-ceph-common-19.1.0-1382.gde09 271 kB/s | 136 kB 00:00 2024-08-19T01:21:35.345 INFO:teuthology.orchestra.run.smithi161.stdout:(21/150): python3-cephfs-19.1.0-1382.gde09dd01. 2.2 MB/s | 188 kB 00:00 2024-08-19T01:21:35.408 INFO:teuthology.orchestra.run.smithi064.stdout:(6/150): ceph-mgr-19.1.0-1382.gde09dd01.el9.x86 5.9 MB/s | 1.5 MB 00:00 2024-08-19T01:21:35.445 INFO:teuthology.orchestra.run.smithi161.stdout:(22/150): librgw2-19.1.0-1382.gde09dd01.el9.x86 6.6 MB/s | 5.4 MB 00:00 2024-08-19T01:21:35.470 INFO:teuthology.orchestra.run.smithi161.stdout:(23/150): python3-rados-19.1.0-1382.gde09dd01.e 2.9 MB/s | 369 kB 00:00 2024-08-19T01:21:35.571 INFO:teuthology.orchestra.run.smithi161.stdout:(24/150): python3-rgw-19.1.0-1382.gde09dd01.el9 1.1 MB/s | 117 kB 00:00 2024-08-19T01:21:35.596 INFO:teuthology.orchestra.run.smithi161.stdout:(25/150): python3-rbd-19.1.0-1382.gde09dd01.el9 2.3 MB/s | 350 kB 00:00 2024-08-19T01:21:35.646 INFO:teuthology.orchestra.run.smithi161.stdout:(26/150): rbd-fuse-19.1.0-1382.gde09dd01.el9.x8 1.1 MB/s | 85 kB 00:00 2024-08-19T01:21:35.700 INFO:teuthology.orchestra.run.smithi064.stdout:(7/150): ceph-mon-19.1.0-1382.gde09dd01.el9.x86 9.6 MB/s | 4.7 MB 00:00 2024-08-19T01:21:35.721 INFO:teuthology.orchestra.run.smithi161.stdout:(27/150): rbd-nbd-19.1.0-1382.gde09dd01.el9.x86 2.2 MB/s | 170 kB 00:00 2024-08-19T01:21:36.017 INFO:teuthology.orchestra.run.smithi064.stdout:(8/150): ceph-common-19.1.0-1382.gde09dd01.el9. 13 MB/s | 21 MB 00:01 2024-08-19T01:21:36.068 INFO:teuthology.orchestra.run.smithi064.stdout:(9/150): ceph-selinux-19.1.0-1382.gde09dd01.el9 506 kB/s | 25 kB 00:00 2024-08-19T01:21:36.172 INFO:teuthology.orchestra.run.smithi161.stdout:(28/150): ceph-test-19.1.0-1382.gde09dd01.el9.x 20 MB/s | 49 MB 00:02 2024-08-19T01:21:36.197 INFO:teuthology.orchestra.run.smithi161.stdout:(29/150): ceph-grafana-dashboards-19.1.0-1382.g 67 kB/s | 32 kB 00:00 2024-08-19T01:21:36.256 INFO:teuthology.orchestra.run.smithi161.stdout:(30/150): rbd-mirror-19.1.0-1382.gde09dd01.el9. 4.7 MB/s | 3.1 MB 00:00 2024-08-19T01:21:36.281 INFO:teuthology.orchestra.run.smithi161.stdout:(31/150): ceph-mgr-cephadm-19.1.0-1382.gde09dd0 1.3 MB/s | 146 kB 00:00 2024-08-19T01:21:36.364 INFO:teuthology.orchestra.run.smithi161.stdout:(32/150): ceph-mgr-modules-core-19.1.0-1382.gde 2.9 MB/s | 249 kB 00:00 2024-08-19T01:21:36.440 INFO:teuthology.orchestra.run.smithi161.stdout:(33/150): ceph-mgr-rook-19.1.0-1382.gde09dd01.e 657 kB/s | 49 kB 00:00 2024-08-19T01:21:36.493 INFO:teuthology.orchestra.run.smithi064.stdout:(10/150): ceph-radosgw-19.1.0-1382.gde09dd01.el 13 MB/s | 11 MB 00:00 2024-08-19T01:21:36.515 INFO:teuthology.orchestra.run.smithi161.stdout:(34/150): ceph-mgr-dashboard-19.1.0-1382.gde09d 11 MB/s | 3.6 MB 00:00 2024-08-19T01:21:36.540 INFO:teuthology.orchestra.run.smithi161.stdout:(35/150): ceph-prometheus-alerts-19.1.0-1382.gd 164 kB/s | 16 kB 00:00 2024-08-19T01:21:36.543 INFO:teuthology.orchestra.run.smithi064.stdout:(11/150): cephfs-mirror-19.1.0-1382.gde09dd01.e 4.4 MB/s | 225 kB 00:00 2024-08-19T01:21:36.582 INFO:teuthology.orchestra.run.smithi161.stdout:(36/150): ceph-volume-19.1.0-1382.gde09dd01.el9 4.2 MB/s | 285 kB 00:00 2024-08-19T01:21:36.602 INFO:teuthology.orchestra.run.smithi064.stdout:(12/150): cephfs-shell-19.1.0-1382.gde09dd01.el 464 kB/s | 27 kB 00:00 2024-08-19T01:21:36.641 INFO:teuthology.orchestra.run.smithi161.stdout:(37/150): cephadm-19.1.0-1382.gde09dd01.el9.noa 7.5 MB/s | 766 kB 00:00 2024-08-19T01:21:36.685 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-19T01:21:36.686 INFO:teuthology.orchestra.run.smithi161.stdout:(38/150): cephfs-top-19.1.0-1382.gde09dd01.el9. 177 kB/s | 18 kB 00:00 2024-08-19T01:21:36.686 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-19T01:21:36.697 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.48.2-2.el9.x86_64 is already installed. 2024-08-19T01:21:36.698 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-19T01:21:36.699 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-19T01:21:36.700 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-19T01:21:36.701 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-19T01:21:36.702 INFO:teuthology.orchestra.run.smithi064.stdout:(13/150): libcephfs-devel-19.1.0-1382.gde09dd01 318 kB/s | 32 kB 00:00 2024-08-19T01:21:36.704 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-19T01:21:36.705 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-6.4.0-3.el9.x86_64 is already installed. 2024-08-19T01:21:36.764 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-08-19T01:21:36.778 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 6.6 k 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 896 k 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 145 k 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 1.5 M 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 146 k 2024-08-19T01:21:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 3.6 M 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 7.4 M 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 49 k 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 11 M 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 49 M 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 285 k 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 766 k 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 225 k 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 27 k 2024-08-19T01:21:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 18 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 32 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 753 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 127 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-19T01:21:36.781 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 188 k 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 369 k 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 350 k 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 117 k 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 85 k 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.1 M 2024-08-19T01:21:36.782 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 170 k 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.4 M 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 3.2 M 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 21 M 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 32 k 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 2.4 M 2024-08-19T01:21:36.783 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 249 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 4.7 M 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 17 M 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1382.gde09dd01.el9 ceph-noarch 16 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 25 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T01:21:36.784 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 163 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 507 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 5.4 M 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T01:21:36.785 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T01:21:36.786 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 45 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:19.1.0-1382.gde09dd01.el9 ceph 136 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T01:21:36.787 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T01:21:36.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-19T01:21:36.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T01:21:36.790 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-08-19T01:21:36.791 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-08-19T01:21:36.792 INFO:teuthology.orchestra.run.smithi123.stdout:Install 148 Packages 2024-08-19T01:21:36.792 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 2 Packages 2024-08-19T01:21:36.792 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:21:36.792 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 218 M 2024-08-19T01:21:36.792 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-08-19T01:21:36.802 INFO:teuthology.orchestra.run.smithi064.stdout:(14/150): libcephfs2-19.1.0-1382.gde09dd01.el9. 7.4 MB/s | 753 kB 00:00 2024-08-19T01:21:36.869 INFO:teuthology.orchestra.run.smithi064.stdout:(15/150): libcephsqlite-19.1.0-1382.gde09dd01.e 2.4 MB/s | 163 kB 00:00 2024-08-19T01:21:36.928 INFO:teuthology.orchestra.run.smithi064.stdout:(16/150): librados-devel-19.1.0-1382.gde09dd01. 2.1 MB/s | 127 kB 00:00 2024-08-19T01:21:36.994 INFO:teuthology.orchestra.run.smithi064.stdout:(17/150): libradosstriper1-19.1.0-1382.gde09dd0 7.4 MB/s | 507 kB 00:00 2024-08-19T01:21:37.175 INFO:teuthology.orchestra.run.smithi161.stdout:(39/150): ceph-mgr-diskprediction-local-19.1.0- 8.1 MB/s | 7.4 MB 00:00 2024-08-19T01:21:37.178 INFO:teuthology.orchestra.run.smithi064.stdout:(18/150): ceph-osd-19.1.0-1382.gde09dd01.el9.x8 9.5 MB/s | 17 MB 00:01 2024-08-19T01:21:37.237 INFO:teuthology.orchestra.run.smithi064.stdout:(19/150): python3-ceph-argparse-19.1.0-1382.gde 778 kB/s | 45 kB 00:00 2024-08-19T01:21:37.295 INFO:teuthology.orchestra.run.smithi064.stdout:(20/150): python3-ceph-common-19.1.0-1382.gde09 2.3 MB/s | 136 kB 00:00 2024-08-19T01:21:37.375 INFO:teuthology.orchestra.run.smithi161.stdout:(40/150): fuse-2.9.9-16.el9.x86_64.rpm 109 kB/s | 80 kB 00:00 2024-08-19T01:21:37.404 INFO:teuthology.orchestra.run.smithi064.stdout:(21/150): librgw2-19.1.0-1382.gde09dd01.el9.x86 13 MB/s | 5.4 MB 00:00 2024-08-19T01:21:37.410 INFO:teuthology.orchestra.run.smithi123.stdout:(1/150): ceph-19.1.0-1382.gde09dd01.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-08-19T01:21:37.429 INFO:teuthology.orchestra.run.smithi064.stdout:(22/150): python3-cephfs-19.1.0-1382.gde09dd01. 1.4 MB/s | 188 kB 00:00 2024-08-19T01:21:37.463 INFO:teuthology.orchestra.run.smithi064.stdout:(23/150): python3-rados-19.1.0-1382.gde09dd01.e 6.2 MB/s | 369 kB 00:00 2024-08-19T01:21:37.488 INFO:teuthology.orchestra.run.smithi064.stdout:(24/150): python3-rbd-19.1.0-1382.gde09dd01.el9 5.8 MB/s | 350 kB 00:00 2024-08-19T01:21:37.513 INFO:teuthology.orchestra.run.smithi064.stdout:(25/150): python3-rgw-19.1.0-1382.gde09dd01.el9 2.3 MB/s | 117 kB 00:00 2024-08-19T01:21:37.547 INFO:teuthology.orchestra.run.smithi064.stdout:(26/150): rbd-fuse-19.1.0-1382.gde09dd01.el9.x8 1.4 MB/s | 85 kB 00:00 2024-08-19T01:21:37.622 INFO:teuthology.orchestra.run.smithi064.stdout:(27/150): rbd-nbd-19.1.0-1382.gde09dd01.el9.x86 2.2 MB/s | 170 kB 00:00 2024-08-19T01:21:37.689 INFO:teuthology.orchestra.run.smithi064.stdout:(28/150): ceph-grafana-dashboards-19.1.0-1382.g 478 kB/s | 32 kB 00:00 2024-08-19T01:21:37.781 INFO:teuthology.orchestra.run.smithi064.stdout:(29/150): rbd-mirror-19.1.0-1382.gde09dd01.el9. 12 MB/s | 3.1 MB 00:00 2024-08-19T01:21:37.806 INFO:teuthology.orchestra.run.smithi064.stdout:(30/150): ceph-mgr-cephadm-19.1.0-1382.gde09dd0 1.2 MB/s | 146 kB 00:00 2024-08-19T01:21:37.983 INFO:teuthology.orchestra.run.smithi161.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 242 kB/s | 195 kB 00:00 2024-08-19T01:21:37.993 INFO:teuthology.orchestra.run.smithi123.stdout:(2/150): ceph-fuse-19.1.0-1382.gde09dd01.el9.x8 1.5 MB/s | 896 kB 00:00 2024-08-19T01:21:38.060 INFO:teuthology.orchestra.run.smithi123.stdout:(3/150): ceph-base-19.1.0-1382.gde09dd01.el9.x8 6.6 MB/s | 5.4 MB 00:00 2024-08-19T01:21:38.098 INFO:teuthology.orchestra.run.smithi064.stdout:(31/150): ceph-mgr-dashboard-19.1.0-1382.gde09d 11 MB/s | 3.6 MB 00:00 2024-08-19T01:21:38.102 INFO:teuthology.orchestra.run.smithi123.stdout:(4/150): ceph-immutable-object-cache-19.1.0-138 1.3 MB/s | 145 kB 00:00 2024-08-19T01:21:38.157 INFO:teuthology.orchestra.run.smithi064.stdout:(32/150): ceph-mgr-modules-core-19.1.0-1382.gde 4.2 MB/s | 249 kB 00:00 2024-08-19T01:21:38.217 INFO:teuthology.orchestra.run.smithi161.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 524 kB/s | 803 kB 00:01 2024-08-19T01:21:38.251 INFO:teuthology.orchestra.run.smithi161.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 289 kB/s | 253 kB 00:00 2024-08-19T01:21:38.261 INFO:teuthology.orchestra.run.smithi123.stdout:(5/150): ceph-mds-19.1.0-1382.gde09dd01.el9.x86 12 MB/s | 2.4 MB 00:00 2024-08-19T01:21:38.545 INFO:teuthology.orchestra.run.smithi123.stdout:(6/150): ceph-mon-19.1.0-1382.gde09dd01.el9.x86 17 MB/s | 4.7 MB 00:00 2024-08-19T01:21:38.591 INFO:teuthology.orchestra.run.smithi064.stdout:(33/150): ceph-test-19.1.0-1382.gde09dd01.el9.x 19 MB/s | 49 MB 00:02 2024-08-19T01:21:38.616 INFO:teuthology.orchestra.run.smithi064.stdout:(34/150): ceph-mgr-rook-19.1.0-1382.gde09dd01.e 107 kB/s | 49 kB 00:00 2024-08-19T01:21:38.636 INFO:teuthology.orchestra.run.smithi123.stdout:(7/150): ceph-mgr-19.1.0-1382.gde09dd01.el9.x86 2.9 MB/s | 1.5 MB 00:00 2024-08-19T01:21:38.666 INFO:teuthology.orchestra.run.smithi064.stdout:(35/150): ceph-prometheus-alerts-19.1.0-1382.gd 220 kB/s | 16 kB 00:00 2024-08-19T01:21:38.691 INFO:teuthology.orchestra.run.smithi064.stdout:(36/150): ceph-volume-19.1.0-1382.gde09dd01.el9 3.7 MB/s | 285 kB 00:00 2024-08-19T01:21:38.733 INFO:teuthology.orchestra.run.smithi064.stdout:(37/150): cephfs-top-19.1.0-1382.gde09dd01.el9. 436 kB/s | 18 kB 00:00 2024-08-19T01:21:38.826 INFO:teuthology.orchestra.run.smithi064.stdout:(38/150): ceph-mgr-diskprediction-local-19.1.0- 7.3 MB/s | 7.4 MB 00:01 2024-08-19T01:21:38.859 INFO:teuthology.orchestra.run.smithi064.stdout:(39/150): cephadm-19.1.0-1382.gde09dd01.el9.noa 3.9 MB/s | 766 kB 00:00 2024-08-19T01:21:38.870 INFO:teuthology.orchestra.run.smithi123.stdout:(8/150): ceph-common-19.1.0-1382.gde09dd01.el9. 13 MB/s | 21 MB 00:01 2024-08-19T01:21:38.912 INFO:teuthology.orchestra.run.smithi123.stdout:(9/150): ceph-selinux-19.1.0-1382.gde09dd01.el9 608 kB/s | 25 kB 00:00 2024-08-19T01:21:38.917 INFO:teuthology.orchestra.run.smithi161.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 154 kB/s | 106 kB 00:00 2024-08-19T01:21:38.951 INFO:teuthology.orchestra.run.smithi161.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 193 kB/s | 135 kB 00:00 2024-08-19T01:21:38.976 INFO:teuthology.orchestra.run.smithi064.stdout:(40/150): fuse-2.9.9-16.el9.x86_64.rpm 330 kB/s | 80 kB 00:00 2024-08-19T01:21:39.085 INFO:teuthology.orchestra.run.smithi064.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-08-19T01:21:39.110 INFO:teuthology.orchestra.run.smithi064.stdout:(42/150): libquadmath-11.4.1-3.el9.x86_64.rpm 780 kB/s | 195 kB 00:00 2024-08-19T01:21:39.118 INFO:teuthology.orchestra.run.smithi161.stdout:(46/150): python3-cryptography-36.0.1-4.el9.x86 1.1 MB/s | 1.2 MB 00:01 2024-08-19T01:21:39.135 INFO:teuthology.orchestra.run.smithi064.stdout:(43/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-08-19T01:21:39.169 INFO:teuthology.orchestra.run.smithi064.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-08-19T01:21:39.194 INFO:teuthology.orchestra.run.smithi064.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-08-19T01:21:39.228 INFO:teuthology.orchestra.run.smithi064.stdout:(46/150): python3-cryptography-36.0.1-4.el9.x86 8.6 MB/s | 1.2 MB 00:00 2024-08-19T01:21:39.253 INFO:teuthology.orchestra.run.smithi064.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-08-19T01:21:39.278 INFO:teuthology.orchestra.run.smithi064.stdout:(48/150): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-08-19T01:21:39.377 INFO:teuthology.orchestra.run.smithi161.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-08-19T01:21:39.395 INFO:teuthology.orchestra.run.smithi064.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 917 kB/s | 107 kB 00:00 2024-08-19T01:21:39.404 INFO:teuthology.orchestra.run.smithi123.stdout:(10/150): ceph-osd-19.1.0-1382.gde09dd01.el9.x8 20 MB/s | 17 MB 00:00 2024-08-19T01:21:39.429 INFO:teuthology.orchestra.run.smithi064.stdout:(50/150): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-08-19T01:21:39.454 INFO:teuthology.orchestra.run.smithi064.stdout:(51/150): elfutils-libelf-devel-0.191-4.el9.x86 372 kB/s | 22 kB 00:00 2024-08-19T01:21:39.454 INFO:teuthology.orchestra.run.smithi123.stdout:(11/150): cephfs-mirror-19.1.0-1382.gde09dd01.e 4.4 MB/s | 225 kB 00:00 2024-08-19T01:21:39.487 INFO:teuthology.orchestra.run.smithi064.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 943 kB 00:00 2024-08-19T01:21:39.502 INFO:teuthology.orchestra.run.smithi161.stdout:(48/150): bison-3.7.4-5.el9.x86_64.rpm 7.4 MB/s | 943 kB 00:00 2024-08-19T01:21:39.513 INFO:teuthology.orchestra.run.smithi123.stdout:(12/150): cephfs-shell-19.1.0-1382.gde09dd01.el 465 kB/s | 27 kB 00:00 2024-08-19T01:21:39.513 INFO:teuthology.orchestra.run.smithi064.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-08-19T01:21:39.538 INFO:teuthology.orchestra.run.smithi064.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-08-19T01:21:39.563 INFO:teuthology.orchestra.run.smithi123.stdout:(13/150): libcephfs-devel-19.1.0-1382.gde09dd01 636 kB/s | 32 kB 00:00 2024-08-19T01:21:39.563 INFO:teuthology.orchestra.run.smithi064.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-08-19T01:21:39.588 INFO:teuthology.orchestra.run.smithi064.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-08-19T01:21:39.594 INFO:teuthology.orchestra.run.smithi161.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-08-19T01:21:39.614 INFO:teuthology.orchestra.run.smithi064.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-08-19T01:21:39.619 INFO:teuthology.orchestra.run.smithi161.stdout:(50/150): python3-requests-2.25.1-8.el9.noarch. 178 kB/s | 125 kB 00:00 2024-08-19T01:21:39.639 INFO:teuthology.orchestra.run.smithi064.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-08-19T01:21:39.644 INFO:teuthology.orchestra.run.smithi161.stdout:(51/150): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-08-19T01:21:39.655 INFO:teuthology.orchestra.run.smithi123.stdout:(14/150): libcephfs2-19.1.0-1382.gde09dd01.el9. 8.0 MB/s | 753 kB 00:00 2024-08-19T01:21:39.672 INFO:teuthology.orchestra.run.smithi064.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-08-19T01:21:39.703 INFO:teuthology.orchestra.run.smithi161.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-08-19T01:21:39.723 INFO:teuthology.orchestra.run.smithi064.stdout:(60/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-08-19T01:21:39.730 INFO:teuthology.orchestra.run.smithi123.stdout:(15/150): libcephsqlite-19.1.0-1382.gde09dd01.e 2.1 MB/s | 163 kB 00:00 2024-08-19T01:21:39.748 INFO:teuthology.orchestra.run.smithi064.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-08-19T01:21:39.781 INFO:teuthology.orchestra.run.smithi064.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 814 kB/s | 47 kB 00:00 2024-08-19T01:21:39.797 INFO:teuthology.orchestra.run.smithi123.stdout:(16/150): librados-devel-19.1.0-1382.gde09dd01. 1.9 MB/s | 127 kB 00:00 2024-08-19T01:21:39.807 INFO:teuthology.orchestra.run.smithi064.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-08-19T01:21:39.820 INFO:teuthology.orchestra.run.smithi161.stdout:(53/150): python3-urllib3-1.26.5-6.el9.noarch.r 247 kB/s | 215 kB 00:00 2024-08-19T01:21:39.841 INFO:teuthology.orchestra.run.smithi064.stdout:(64/150): libtool-2.4.6-46.el9.x86_64.rpm 2.8 MB/s | 581 kB 00:00 2024-08-19T01:21:39.853 INFO:teuthology.orchestra.run.smithi161.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 1.3 MB/s | 313 kB 00:00 2024-08-19T01:21:39.866 INFO:teuthology.orchestra.run.smithi064.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-08-19T01:21:39.882 INFO:teuthology.orchestra.run.smithi123.stdout:(17/150): libradosstriper1-19.1.0-1382.gde09dd0 5.9 MB/s | 507 kB 00:00 2024-08-19T01:21:39.891 INFO:teuthology.orchestra.run.smithi064.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-08-19T01:21:39.904 INFO:teuthology.orchestra.run.smithi161.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-08-19T01:21:39.937 INFO:teuthology.orchestra.run.smithi161.stdout:(56/150): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 18 kB 00:00 2024-08-19T01:21:39.958 INFO:teuthology.orchestra.run.smithi064.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-08-19T01:21:39.971 INFO:teuthology.orchestra.run.smithi161.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-08-19T01:21:40.021 INFO:teuthology.orchestra.run.smithi161.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2024-08-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi064.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-08-19T01:21:40.039 INFO:teuthology.orchestra.run.smithi123.stdout:(18/150): ceph-radosgw-19.1.0-1382.gde09dd01.el 7.5 MB/s | 11 MB 00:01 2024-08-19T01:21:40.081 INFO:teuthology.orchestra.run.smithi123.stdout:(19/150): python3-ceph-argparse-19.1.0-1382.gde 1.1 MB/s | 45 kB 00:00 2024-08-19T01:21:40.088 INFO:teuthology.orchestra.run.smithi161.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-08-19T01:21:40.092 INFO:teuthology.orchestra.run.smithi064.stdout:(69/150): openssl-devel-3.2.2-2.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-08-19T01:21:40.117 INFO:teuthology.orchestra.run.smithi064.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-08-19T01:21:40.143 INFO:teuthology.orchestra.run.smithi064.stdout:(71/150): postgresql-contrib-13.14-1.el9.x86_64 4.8 MB/s | 911 kB 00:00 2024-08-19T01:21:40.147 INFO:teuthology.orchestra.run.smithi161.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 438 kB/s | 25 kB 00:00 2024-08-19T01:21:40.172 INFO:teuthology.orchestra.run.smithi161.stdout:(61/150): libtool-2.4.6-46.el9.x86_64.rpm 3.8 MB/s | 581 kB 00:00 2024-08-19T01:21:40.173 INFO:teuthology.orchestra.run.smithi123.stdout:(20/150): librgw2-19.1.0-1382.gde09dd01.el9.x86 18 MB/s | 5.4 MB 00:00 2024-08-19T01:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-08-19T01:21:40.198 INFO:teuthology.orchestra.run.smithi161.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-08-19T01:21:40.198 INFO:teuthology.orchestra.run.smithi123.stdout:(21/150): python3-ceph-common-19.1.0-1382.gde09 1.1 MB/s | 136 kB 00:00 2024-08-19T01:21:40.201 INFO:teuthology.orchestra.run.smithi064.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-08-19T01:21:40.223 INFO:teuthology.orchestra.run.smithi123.stdout:(22/150): python3-cephfs-19.1.0-1382.gde09dd01. 3.7 MB/s | 188 kB 00:00 2024-08-19T01:21:40.239 INFO:teuthology.orchestra.run.smithi161.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-08-19T01:21:40.243 INFO:teuthology.orchestra.run.smithi064.stdout:(74/150): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-08-19T01:21:40.257 INFO:teuthology.orchestra.run.smithi123.stdout:(23/150): python3-rados-19.1.0-1382.gde09dd01.e 6.2 MB/s | 369 kB 00:00 2024-08-19T01:21:40.264 INFO:teuthology.orchestra.run.smithi161.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-08-19T01:21:40.282 INFO:teuthology.orchestra.run.smithi123.stdout:(24/150): python3-rbd-19.1.0-1382.gde09dd01.el9 5.8 MB/s | 350 kB 00:00 2024-08-19T01:21:40.298 INFO:teuthology.orchestra.run.smithi161.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-08-19T01:21:40.307 INFO:teuthology.orchestra.run.smithi123.stdout:(25/150): python3-rgw-19.1.0-1382.gde09dd01.el9 2.3 MB/s | 117 kB 00:00 2024-08-19T01:21:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:(75/150): postgresql-server-13.14-1.el9.x86_64. 25 MB/s | 5.8 MB 00:00 2024-08-19T01:21:40.333 INFO:teuthology.orchestra.run.smithi123.stdout:(26/150): rbd-fuse-19.1.0-1382.gde09dd01.el9.x8 1.6 MB/s | 85 kB 00:00 2024-08-19T01:21:40.352 INFO:teuthology.orchestra.run.smithi064.stdout:(76/150): python3-devel-3.9.19-4.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-08-19T01:21:40.377 INFO:teuthology.orchestra.run.smithi064.stdout:(77/150): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-08-19T01:21:40.392 INFO:teuthology.orchestra.run.smithi123.stdout:(27/150): rbd-nbd-19.1.0-1382.gde09dd01.el9.x86 2.9 MB/s | 170 kB 00:00 2024-08-19T01:21:40.403 INFO:teuthology.orchestra.run.smithi064.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-08-19T01:21:40.420 INFO:teuthology.orchestra.run.smithi064.stdout:(79/150): python3-jsonpointer-2.0-4.el9.noarch. 457 kB/s | 19 kB 00:00 2024-08-19T01:21:40.445 INFO:teuthology.orchestra.run.smithi064.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-08-19T01:21:40.462 INFO:teuthology.orchestra.run.smithi064.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 828 kB/s | 35 kB 00:00 2024-08-19T01:21:40.476 INFO:teuthology.orchestra.run.smithi123.stdout:(28/150): ceph-grafana-dashboards-19.1.0-1382.g 382 kB/s | 32 kB 00:00 2024-08-19T01:21:40.537 INFO:teuthology.orchestra.run.smithi064.stdout:(82/150): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-08-19T01:21:40.562 INFO:teuthology.orchestra.run.smithi064.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-08-19T01:21:40.588 INFO:teuthology.orchestra.run.smithi064.stdout:(84/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-08-19T01:21:40.613 INFO:teuthology.orchestra.run.smithi064.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-08-19T01:21:40.638 INFO:teuthology.orchestra.run.smithi064.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-08-19T01:21:40.663 INFO:teuthology.orchestra.run.smithi064.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-08-19T01:21:40.680 INFO:teuthology.orchestra.run.smithi064.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.7 MB/s | 159 kB 00:00 2024-08-19T01:21:40.714 INFO:teuthology.orchestra.run.smithi064.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-08-19T01:21:40.739 INFO:teuthology.orchestra.run.smithi064.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-08-19T01:21:40.740 INFO:teuthology.orchestra.run.smithi161.stdout:(66/150): openblas-openmp-0.3.21-2.el9.x86_64.r 10 MB/s | 4.7 MB 00:00 2024-08-19T01:21:40.765 INFO:teuthology.orchestra.run.smithi064.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-08-19T01:21:40.824 INFO:teuthology.orchestra.run.smithi161.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-08-19T01:21:40.848 INFO:teuthology.orchestra.run.smithi064.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-08-19T01:21:40.873 INFO:teuthology.orchestra.run.smithi064.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 384 kB/s | 42 kB 00:00 2024-08-19T01:21:40.898 INFO:teuthology.orchestra.run.smithi064.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-08-19T01:21:40.907 INFO:teuthology.orchestra.run.smithi161.stdout:(68/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-08-19T01:21:40.916 INFO:teuthology.orchestra.run.smithi064.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 221 kB 00:00 2024-08-19T01:21:40.926 INFO:teuthology.orchestra.run.smithi123.stdout:(29/150): ceph-test-19.1.0-1382.gde09dd01.el9.x 24 MB/s | 49 MB 00:02 2024-08-19T01:21:40.941 INFO:teuthology.orchestra.run.smithi161.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 7.6 MB/s | 911 kB 00:00 2024-08-19T01:21:40.941 INFO:teuthology.orchestra.run.smithi064.stdout:(96/150): qatlib-service-24.02.0-1.el9.x86_64.r 857 kB/s | 36 kB 00:00 2024-08-19T01:21:40.966 INFO:teuthology.orchestra.run.smithi064.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-08-19T01:21:40.976 INFO:teuthology.orchestra.run.smithi123.stdout:(30/150): rbd-mirror-19.1.0-1382.gde09dd01.el9. 4.6 MB/s | 3.1 MB 00:00 2024-08-19T01:21:40.991 INFO:teuthology.orchestra.run.smithi064.stdout:(98/150): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-08-19T01:21:41.001 INFO:teuthology.orchestra.run.smithi123.stdout:(31/150): ceph-mgr-cephadm-19.1.0-1382.gde09dd0 278 kB/s | 146 kB 00:00 2024-08-19T01:21:41.008 INFO:teuthology.orchestra.run.smithi161.stdout:(70/150): openssl-devel-3.2.2-2.el9.x86_64.rpm 6.3 MB/s | 4.4 MB 00:00 2024-08-19T01:21:41.017 INFO:teuthology.orchestra.run.smithi064.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-08-19T01:21:41.036 INFO:teuthology.orchestra.run.smithi064.stdout:(100/150): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 2.1 MB/s | 96 kB 00:00 2024-08-19T01:21:41.058 INFO:teuthology.orchestra.run.smithi161.stdout:(71/150): postgresql-private-libs-13.14-1.el9.x 887 kB/s | 133 kB 00:00 2024-08-19T01:21:41.061 INFO:teuthology.orchestra.run.smithi064.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.4 MB/s | 64 kB 00:00 2024-08-19T01:21:41.085 INFO:teuthology.orchestra.run.smithi123.stdout:(32/150): ceph-mgr-dashboard-19.1.0-1382.gde09d 22 MB/s | 3.6 MB 00:00 2024-08-19T01:21:41.086 INFO:teuthology.orchestra.run.smithi064.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-08-19T01:21:41.091 INFO:teuthology.orchestra.run.smithi161.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-08-19T01:21:41.110 INFO:teuthology.orchestra.run.smithi123.stdout:(33/150): ceph-mgr-modules-core-19.1.0-1382.gde 2.2 MB/s | 249 kB 00:00 2024-08-19T01:21:41.135 INFO:teuthology.orchestra.run.smithi123.stdout:(34/150): ceph-mgr-rook-19.1.0-1382.gde09dd01.e 987 kB/s | 49 kB 00:00 2024-08-19T01:21:41.161 INFO:teuthology.orchestra.run.smithi123.stdout:(35/150): ceph-prometheus-alerts-19.1.0-1382.gd 329 kB/s | 16 kB 00:00 2024-08-19T01:21:41.175 INFO:teuthology.orchestra.run.smithi161.stdout:(73/150): postgresql-server-13.14-1.el9.x86_64. 25 MB/s | 5.8 MB 00:00 2024-08-19T01:21:41.178 INFO:teuthology.orchestra.run.smithi064.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-08-19T01:21:41.186 INFO:teuthology.orchestra.run.smithi123.stdout:(36/150): ceph-volume-19.1.0-1382.gde09dd01.el9 5.6 MB/s | 285 kB 00:00 2024-08-19T01:21:41.200 INFO:teuthology.orchestra.run.smithi161.stdout:(74/150): python3-attrs-20.3.0-7.el9.noarch.rpm 615 kB/s | 87 kB 00:00 2024-08-19T01:21:41.217 INFO:teuthology.orchestra.run.smithi161.stdout:(75/150): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-08-19T01:21:41.236 INFO:teuthology.orchestra.run.smithi123.stdout:(37/150): cephfs-top-19.1.0-1382.gde09dd01.el9. 366 kB/s | 18 kB 00:00 2024-08-19T01:21:41.270 INFO:teuthology.orchestra.run.smithi123.stdout:(38/150): cephadm-19.1.0-1382.gde09dd01.el9.noa 6.9 MB/s | 766 kB 00:00 2024-08-19T01:21:41.285 INFO:teuthology.orchestra.run.smithi161.stdout:(76/150): python3-jinja2-2.11.3-6.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-08-19T01:21:41.337 INFO:teuthology.orchestra.run.smithi064.stdout:(104/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-08-19T01:21:41.368 INFO:teuthology.orchestra.run.smithi161.stdout:(77/150): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-08-19T01:21:41.385 INFO:teuthology.orchestra.run.smithi161.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-08-19T01:21:41.387 INFO:teuthology.orchestra.run.smithi123.stdout:(39/150): ceph-mgr-diskprediction-local-19.1.0- 18 MB/s | 7.4 MB 00:00 2024-08-19T01:21:41.395 INFO:teuthology.orchestra.run.smithi064.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.5 MB/s | 548 kB 00:00 2024-08-19T01:21:41.403 INFO:teuthology.orchestra.run.smithi123.stdout:(40/150): fuse-2.9.9-16.el9.x86_64.rpm 478 kB/s | 80 kB 00:00 2024-08-19T01:21:41.410 INFO:teuthology.orchestra.run.smithi161.stdout:(79/150): python3-devel-3.9.19-4.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-08-19T01:21:41.421 INFO:teuthology.orchestra.run.smithi064.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-08-19T01:21:41.436 INFO:teuthology.orchestra.run.smithi161.stdout:(80/150): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-08-19T01:21:41.446 INFO:teuthology.orchestra.run.smithi064.stdout:(107/150): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-08-19T01:21:41.461 INFO:teuthology.orchestra.run.smithi161.stdout:(81/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-08-19T01:21:41.480 INFO:teuthology.orchestra.run.smithi064.stdout:(108/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-08-19T01:21:41.486 INFO:teuthology.orchestra.run.smithi161.stdout:(82/150): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-08-19T01:21:41.487 INFO:teuthology.orchestra.run.smithi123.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-08-19T01:21:41.505 INFO:teuthology.orchestra.run.smithi064.stdout:(109/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-08-19T01:21:41.513 INFO:teuthology.orchestra.run.smithi123.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-08-19T01:21:41.536 INFO:teuthology.orchestra.run.smithi161.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.2 MB/s | 250 kB 00:00 2024-08-19T01:21:41.537 INFO:teuthology.orchestra.run.smithi123.stdout:(43/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-08-19T01:21:41.538 INFO:teuthology.orchestra.run.smithi064.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-08-19T01:21:41.563 INFO:teuthology.orchestra.run.smithi123.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-08-19T01:21:41.572 INFO:teuthology.orchestra.run.smithi064.stdout:(111/150): luarocks-3.9.2-5.el9.noarch.rpm 4.7 MB/s | 151 kB 00:00 2024-08-19T01:21:41.588 INFO:teuthology.orchestra.run.smithi123.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-08-19T01:21:41.597 INFO:teuthology.orchestra.run.smithi064.stdout:(112/150): liboath-2.6.7-2.el9.x86_64.rpm 414 kB/s | 49 kB 00:00 2024-08-19T01:21:41.603 INFO:teuthology.orchestra.run.smithi161.stdout:(84/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 5.1 MB 00:00 2024-08-19T01:21:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-08-19T01:21:41.628 INFO:teuthology.orchestra.run.smithi161.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 841 kB/s | 77 kB 00:00 2024-08-19T01:21:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:(47/150): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-08-19T01:21:41.648 INFO:teuthology.orchestra.run.smithi064.stdout:(113/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-08-19T01:21:41.654 INFO:teuthology.orchestra.run.smithi161.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-08-19T01:21:41.673 INFO:teuthology.orchestra.run.smithi064.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-08-19T01:21:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:(48/150): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-08-19T01:21:41.687 INFO:teuthology.orchestra.run.smithi161.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-08-19T01:21:41.698 INFO:teuthology.orchestra.run.smithi064.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-08-19T01:21:41.713 INFO:teuthology.orchestra.run.smithi161.stdout:(88/150): python3-oauthlib-3.1.1-5.el9.noarch.r 982 kB/s | 222 kB 00:00 2024-08-19T01:21:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 6.7 MB/s | 689 kB 00:00 2024-08-19T01:21:41.738 INFO:teuthology.orchestra.run.smithi161.stdout:(89/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 159 kB 00:00 2024-08-19T01:21:41.740 INFO:teuthology.orchestra.run.smithi123.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-08-19T01:21:41.740 INFO:teuthology.orchestra.run.smithi064.stdout:(116/150): python3-certifi-2023.05.07-4.el9.noa 339 kB/s | 14 kB 00:00 2024-08-19T01:21:41.763 INFO:teuthology.orchestra.run.smithi161.stdout:(90/150): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-08-19T01:21:41.765 INFO:teuthology.orchestra.run.smithi123.stdout:(51/150): elfutils-libelf-devel-0.191-4.el9.x86 433 kB/s | 22 kB 00:00 2024-08-19T01:21:41.788 INFO:teuthology.orchestra.run.smithi161.stdout:(91/150): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-08-19T01:21:41.790 INFO:teuthology.orchestra.run.smithi123.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-08-19T01:21:41.801 INFO:teuthology.orchestra.run.smithi064.stdout:(117/150): python3-asyncssh-2.13.2-5.el9.noarch 2.6 MB/s | 548 kB 00:00 2024-08-19T01:21:41.814 INFO:teuthology.orchestra.run.smithi161.stdout:(92/150): python3-requests-oauthlib-1.3.0-12.el 712 kB/s | 54 kB 00:00 2024-08-19T01:21:41.815 INFO:teuthology.orchestra.run.smithi123.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 650 kB/s | 33 kB 00:00 2024-08-19T01:21:41.832 INFO:teuthology.orchestra.run.smithi064.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-08-19T01:21:41.849 INFO:teuthology.orchestra.run.smithi123.stdout:(54/150): bison-3.7.4-5.el9.x86_64.rpm 4.4 MB/s | 943 kB 00:00 2024-08-19T01:21:41.864 INFO:teuthology.orchestra.run.smithi161.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 554 kB/s | 42 kB 00:00 2024-08-19T01:21:41.866 INFO:teuthology.orchestra.run.smithi064.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 5.5 MB/s | 358 kB 00:00 2024-08-19T01:21:41.874 INFO:teuthology.orchestra.run.smithi123.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-08-19T01:21:41.889 INFO:teuthology.orchestra.run.smithi161.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 586 kB/s | 44 kB 00:00 2024-08-19T01:21:41.891 INFO:teuthology.orchestra.run.smithi064.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-08-19T01:21:41.900 INFO:teuthology.orchestra.run.smithi123.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-08-19T01:21:41.916 INFO:teuthology.orchestra.run.smithi064.stdout:(121/150): python3-google-auth-2.33.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-08-19T01:21:41.927 INFO:teuthology.orchestra.run.smithi123.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.0 MB/s | 160 kB 00:00 2024-08-19T01:21:41.948 INFO:teuthology.orchestra.run.smithi161.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 615 kB/s | 36 kB 00:00 2024-08-19T01:21:41.950 INFO:teuthology.orchestra.run.smithi064.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-08-19T01:21:41.952 INFO:teuthology.orchestra.run.smithi123.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 865 kB/s | 45 kB 00:00 2024-08-19T01:21:41.975 INFO:teuthology.orchestra.run.smithi064.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 430 kB/s | 11 kB 00:00 2024-08-19T01:21:41.986 INFO:teuthology.orchestra.run.smithi123.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-08-19T01:21:42.000 INFO:teuthology.orchestra.run.smithi064.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 710 kB/s | 18 kB 00:00 2024-08-19T01:21:42.019 INFO:teuthology.orchestra.run.smithi123.stdout:(60/150): libtool-2.4.6-46.el9.x86_64.rpm 8.5 MB/s | 581 kB 00:00 2024-08-19T01:21:42.026 INFO:teuthology.orchestra.run.smithi064.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 930 kB/s | 23 kB 00:00 2024-08-19T01:21:42.045 INFO:teuthology.orchestra.run.smithi123.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-08-19T01:21:42.051 INFO:teuthology.orchestra.run.smithi064.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 780 kB/s | 19 kB 00:00 2024-08-19T01:21:42.070 INFO:teuthology.orchestra.run.smithi123.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-08-19T01:21:42.076 INFO:teuthology.orchestra.run.smithi064.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 787 kB/s | 20 kB 00:00 2024-08-19T01:21:42.095 INFO:teuthology.orchestra.run.smithi123.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-08-19T01:21:42.120 INFO:teuthology.orchestra.run.smithi123.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-08-19T01:21:42.146 INFO:teuthology.orchestra.run.smithi123.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-08-19T01:21:42.152 INFO:teuthology.orchestra.run.smithi064.stdout:(128/150): libarrow-9.0.0-11.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-08-19T01:21:42.165 INFO:teuthology.orchestra.run.smithi161.stdout:(96/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-08-19T01:21:42.190 INFO:teuthology.orchestra.run.smithi161.stdout:(97/150): qatlib-24.02.0-1.el9.x86_64.rpm 677 kB/s | 221 kB 00:00 2024-08-19T01:21:42.196 INFO:teuthology.orchestra.run.smithi123.stdout:(66/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.4 MB/s | 3.0 MB 00:00 2024-08-19T01:21:42.202 INFO:teuthology.orchestra.run.smithi064.stdout:(129/150): python3-grpcio-1.46.7-10.el9.x86_64. 6.6 MB/s | 2.0 MB 00:00 2024-08-19T01:21:42.215 INFO:teuthology.orchestra.run.smithi161.stdout:(98/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 177 kB/s | 47 kB 00:00 2024-08-19T01:21:42.227 INFO:teuthology.orchestra.run.smithi064.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-08-19T01:21:42.240 INFO:teuthology.orchestra.run.smithi161.stdout:(99/150): socat-1.7.4.1-6.el9.x86_64.rpm 4.0 MB/s | 304 kB 00:00 2024-08-19T01:21:42.253 INFO:teuthology.orchestra.run.smithi064.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-08-19T01:21:42.263 INFO:teuthology.orchestra.run.smithi123.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-08-19T01:21:42.266 INFO:teuthology.orchestra.run.smithi161.stdout:(100/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-08-19T01:21:42.278 INFO:teuthology.orchestra.run.smithi064.stdout:(132/150): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-08-19T01:21:42.291 INFO:teuthology.orchestra.run.smithi161.stdout:(101/150): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.2 MB/s | 96 kB 00:00 2024-08-19T01:21:42.303 INFO:teuthology.orchestra.run.smithi064.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-08-19T01:21:42.316 INFO:teuthology.orchestra.run.smithi161.stdout:(102/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 842 kB/s | 64 kB 00:00 2024-08-19T01:21:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-08-19T01:21:42.328 INFO:teuthology.orchestra.run.smithi064.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-08-19T01:21:42.341 INFO:teuthology.orchestra.run.smithi161.stdout:(103/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 594 kB/s | 45 kB 00:00 2024-08-19T01:21:42.354 INFO:teuthology.orchestra.run.smithi064.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-08-19T01:21:42.355 INFO:teuthology.orchestra.run.smithi123.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-08-19T01:21:42.379 INFO:teuthology.orchestra.run.smithi064.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-08-19T01:21:42.380 INFO:teuthology.orchestra.run.smithi123.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-08-19T01:21:42.392 INFO:teuthology.orchestra.run.smithi161.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-08-19T01:21:42.404 INFO:teuthology.orchestra.run.smithi064.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-08-19T01:21:42.417 INFO:teuthology.orchestra.run.smithi161.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-08-19T01:21:42.429 INFO:teuthology.orchestra.run.smithi064.stdout:(138/150): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-08-19T01:21:42.439 INFO:teuthology.orchestra.run.smithi123.stdout:(71/150): openssl-devel-3.2.2-2.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-08-19T01:21:42.442 INFO:teuthology.orchestra.run.smithi161.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-08-19T01:21:42.455 INFO:teuthology.orchestra.run.smithi064.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-08-19T01:21:42.480 INFO:teuthology.orchestra.run.smithi064.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-08-19T01:21:42.481 INFO:teuthology.orchestra.run.smithi123.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-08-19T01:21:42.505 INFO:teuthology.orchestra.run.smithi064.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-08-19T01:21:42.506 INFO:teuthology.orchestra.run.smithi123.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-08-19T01:21:42.522 INFO:teuthology.orchestra.run.smithi064.stdout:(142/150): python3-typing-extensions-4.12.2-2.e 1.1 MB/s | 74 kB 00:00 2024-08-19T01:21:42.539 INFO:teuthology.orchestra.run.smithi123.stdout:(74/150): python3-colorama-0.4.6-3.el9.noarch.r 1.8 MB/s | 61 kB 00:00 2024-08-19T01:21:42.547 INFO:teuthology.orchestra.run.smithi064.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-08-19T01:21:42.551 INFO:teuthology.orchestra.run.smithi161.stdout:(107/150): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-08-19T01:21:42.573 INFO:teuthology.orchestra.run.smithi064.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 801 kB/s | 20 kB 00:00 2024-08-19T01:21:42.593 INFO:teuthology.orchestra.run.smithi161.stdout:(108/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 926 kB/s | 26 kB 00:00 2024-08-19T01:21:42.598 INFO:teuthology.orchestra.run.smithi064.stdout:(145/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-08-19T01:21:42.615 INFO:teuthology.orchestra.run.smithi123.stdout:(75/150): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-08-19T01:21:42.618 INFO:teuthology.orchestra.run.smithi161.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-08-19T01:21:42.623 INFO:teuthology.orchestra.run.smithi064.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-08-19T01:21:42.640 INFO:teuthology.orchestra.run.smithi123.stdout:(76/150): python3-devel-3.9.19-4.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-08-19T01:21:42.643 INFO:teuthology.orchestra.run.smithi161.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-08-19T01:21:42.649 INFO:teuthology.orchestra.run.smithi064.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-08-19T01:21:42.665 INFO:teuthology.orchestra.run.smithi123.stdout:(77/150): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-08-19T01:21:42.669 INFO:teuthology.orchestra.run.smithi161.stdout:(111/150): luarocks-3.9.2-5.el9.noarch.rpm 6.2 MB/s | 151 kB 00:00 2024-08-19T01:21:42.690 INFO:teuthology.orchestra.run.smithi064.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-08-19T01:21:42.691 INFO:teuthology.orchestra.run.smithi123.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-08-19T01:21:42.716 INFO:teuthology.orchestra.run.smithi123.stdout:(79/150): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-08-19T01:21:42.727 INFO:teuthology.orchestra.run.smithi161.stdout:(112/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-08-19T01:21:42.741 INFO:teuthology.orchestra.run.smithi123.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-08-19T01:21:42.761 INFO:teuthology.orchestra.run.smithi161.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-08-19T01:21:42.766 INFO:teuthology.orchestra.run.smithi123.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 691 kB/s | 35 kB 00:00 2024-08-19T01:21:42.787 INFO:teuthology.orchestra.run.smithi161.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-08-19T01:21:42.812 INFO:teuthology.orchestra.run.smithi161.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-08-19T01:21:42.837 INFO:teuthology.orchestra.run.smithi161.stdout:(116/150): python3-certifi-2023.05.07-4.el9.noa 565 kB/s | 14 kB 00:00 2024-08-19T01:21:42.842 INFO:teuthology.orchestra.run.smithi123.stdout:(82/150): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-08-19T01:21:42.862 INFO:teuthology.orchestra.run.smithi161.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 6.7 MB/s | 172 kB 00:00 2024-08-19T01:21:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-08-19T01:21:42.892 INFO:teuthology.orchestra.run.smithi123.stdout:(84/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-08-19T01:21:42.896 INFO:teuthology.orchestra.run.smithi161.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-08-19T01:21:42.899 INFO:teuthology.orchestra.run.smithi064.stdout:(149/150): librados2-19.1.0-1382.gde09dd01.el9. 12 MB/s | 3.4 MB 00:00 2024-08-19T01:21:42.913 INFO:teuthology.orchestra.run.smithi161.stdout:(119/150): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-08-19T01:21:42.918 INFO:teuthology.orchestra.run.smithi123.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-08-19T01:21:42.943 INFO:teuthology.orchestra.run.smithi123.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-08-19T01:21:42.947 INFO:teuthology.orchestra.run.smithi161.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 6.0 MB/s | 303 kB 00:00 2024-08-19T01:21:42.968 INFO:teuthology.orchestra.run.smithi123.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-08-19T01:21:42.993 INFO:teuthology.orchestra.run.smithi123.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-08-19T01:21:42.997 INFO:teuthology.orchestra.run.smithi161.stdout:(121/150): python3-google-auth-2.33.0-1.el9.noa 2.6 MB/s | 220 kB 00:00 2024-08-19T01:21:43.030 INFO:teuthology.orchestra.run.smithi161.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-08-19T01:21:43.036 INFO:teuthology.orchestra.run.smithi123.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-08-19T01:21:43.049 INFO:teuthology.orchestra.run.smithi064.stdout:(150/150): librbd1-19.1.0-1382.gde09dd01.el9.x8 7.9 MB/s | 3.2 MB 00:00 2024-08-19T01:21:43.055 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-08-19T01:21:43.055 INFO:teuthology.orchestra.run.smithi064.stdout:Total 24 MB/s | 218 MB 00:09 2024-08-19T01:21:43.056 INFO:teuthology.orchestra.run.smithi161.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-08-19T01:21:43.061 INFO:teuthology.orchestra.run.smithi123.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 762 kB/s | 51 kB 00:00 2024-08-19T01:21:43.081 INFO:teuthology.orchestra.run.smithi161.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 710 kB/s | 18 kB 00:00 2024-08-19T01:21:43.086 INFO:teuthology.orchestra.run.smithi123.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-08-19T01:21:43.106 INFO:teuthology.orchestra.run.smithi161.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 928 kB/s | 23 kB 00:00 2024-08-19T01:21:43.148 INFO:teuthology.orchestra.run.smithi161.stdout:(126/150): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-08-19T01:21:43.161 INFO:teuthology.orchestra.run.smithi123.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-08-19T01:21:43.173 INFO:teuthology.orchestra.run.smithi161.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 289 kB/s | 19 kB 00:00 2024-08-19T01:21:43.187 INFO:teuthology.orchestra.run.smithi123.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-08-19T01:21:43.198 INFO:teuthology.orchestra.run.smithi161.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-08-19T01:21:43.212 INFO:teuthology.orchestra.run.smithi123.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 886 kB/s | 44 kB 00:00 2024-08-19T01:21:43.224 INFO:teuthology.orchestra.run.smithi161.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-08-19T01:21:43.237 INFO:teuthology.orchestra.run.smithi123.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-08-19T01:21:43.249 INFO:teuthology.orchestra.run.smithi161.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-08-19T01:21:43.262 INFO:teuthology.orchestra.run.smithi123.stdout:(96/150): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-08-19T01:21:43.275 INFO:teuthology.orchestra.run.smithi161.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-08-19T01:21:43.288 INFO:teuthology.orchestra.run.smithi123.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-08-19T01:21:43.308 INFO:teuthology.orchestra.run.smithi161.stdout:(132/150): protobuf-compiler-3.14.0-13.el9.x86_ 871 kB/s | 863 kB 00:00 2024-08-19T01:21:43.313 INFO:teuthology.orchestra.run.smithi123.stdout:(98/150): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-08-19T01:21:43.338 INFO:teuthology.orchestra.run.smithi123.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-08-19T01:21:43.342 INFO:teuthology.orchestra.run.smithi161.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-08-19T01:21:43.363 INFO:teuthology.orchestra.run.smithi123.stdout:(100/150): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-08-19T01:21:43.367 INFO:teuthology.orchestra.run.smithi161.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-08-19T01:21:43.389 INFO:teuthology.orchestra.run.smithi123.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-19T01:21:43.394 INFO:teuthology.orchestra.run.smithi161.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-08-19T01:21:43.417 INFO:teuthology.orchestra.run.smithi161.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-08-19T01:21:43.429 INFO:teuthology.orchestra.run.smithi123.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 689 kB/s | 45 kB 00:00 2024-08-19T01:21:43.443 INFO:teuthology.orchestra.run.smithi161.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-08-19T01:21:43.477 INFO:teuthology.orchestra.run.smithi161.stdout:(138/150): python3-kubernetes-26.1.0-2.el9.noar 3.4 MB/s | 1.0 MB 00:00 2024-08-19T01:21:43.502 INFO:teuthology.orchestra.run.smithi161.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-08-19T01:21:43.514 INFO:teuthology.orchestra.run.smithi123.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-08-19T01:21:43.527 INFO:teuthology.orchestra.run.smithi161.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-08-19T01:21:43.552 INFO:teuthology.orchestra.run.smithi161.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-08-19T01:21:43.578 INFO:teuthology.orchestra.run.smithi161.stdout:(142/150): python3-typing-extensions-4.12.2-2.e 990 kB/s | 74 kB 00:00 2024-08-19T01:21:43.603 INFO:teuthology.orchestra.run.smithi161.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-08-19T01:21:43.628 INFO:teuthology.orchestra.run.smithi161.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-19T01:21:43.662 INFO:teuthology.orchestra.run.smithi161.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-08-19T01:21:43.681 INFO:teuthology.orchestra.run.smithi123.stdout:(104/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-08-19T01:21:43.687 INFO:teuthology.orchestra.run.smithi161.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-08-19T01:21:43.712 INFO:teuthology.orchestra.run.smithi161.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-08-19T01:21:43.723 INFO:teuthology.orchestra.run.smithi123.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-08-19T01:21:43.748 INFO:teuthology.orchestra.run.smithi123.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 846 kB/s | 19 kB 00:00 2024-08-19T01:21:43.763 INFO:teuthology.orchestra.run.smithi161.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-19T01:21:43.782 INFO:teuthology.orchestra.run.smithi123.stdout:(107/150): gperftools-libs-2.9.1-2.el9.x86_64.r 3.1 MB/s | 309 kB 00:00 2024-08-19T01:21:43.807 INFO:teuthology.orchestra.run.smithi123.stdout:(108/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-08-19T01:21:43.832 INFO:teuthology.orchestra.run.smithi123.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-08-19T01:21:43.858 INFO:teuthology.orchestra.run.smithi123.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-08-19T01:21:43.883 INFO:teuthology.orchestra.run.smithi123.stdout:(111/150): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-08-19T01:21:43.934 INFO:teuthology.orchestra.run.smithi123.stdout:(112/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-08-19T01:21:43.967 INFO:teuthology.orchestra.run.smithi123.stdout:(113/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.6 MB/s | 863 kB 00:00 2024-08-19T01:21:44.009 INFO:teuthology.orchestra.run.smithi123.stdout:(114/150): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-08-19T01:21:44.013 INFO:teuthology.orchestra.run.smithi161.stdout:(149/150): librbd1-19.1.0-1382.gde09dd01.el9.x8 11 MB/s | 3.2 MB 00:00 2024-08-19T01:21:44.026 INFO:teuthology.orchestra.run.smithi123.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 2.2 MB/s | 32 kB 00:00 2024-08-19T01:21:44.043 INFO:teuthology.orchestra.run.smithi123.stdout:(116/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 576 kB/s | 43 kB 00:00 2024-08-19T01:21:44.060 INFO:teuthology.orchestra.run.smithi123.stdout:(117/150): python3-certifi-2023.05.07-4.el9.noa 419 kB/s | 14 kB 00:00 2024-08-19T01:21:44.071 INFO:teuthology.orchestra.run.smithi161.stdout:(150/150): librados2-19.1.0-1382.gde09dd01.el9. 8.9 MB/s | 3.4 MB 00:00 2024-08-19T01:21:44.078 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-08-19T01:21:44.078 INFO:teuthology.orchestra.run.smithi161.stdout:Total 18 MB/s | 218 MB 00:11 2024-08-19T01:21:44.093 INFO:teuthology.orchestra.run.smithi123.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-08-19T01:21:44.127 INFO:teuthology.orchestra.run.smithi123.stdout:(119/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-08-19T01:21:44.161 INFO:teuthology.orchestra.run.smithi123.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-08-19T01:21:44.166 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-08-19T01:21:44.186 INFO:teuthology.orchestra.run.smithi123.stdout:(121/150): python3-google-auth-2.33.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-08-19T01:21:44.220 INFO:teuthology.orchestra.run.smithi123.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-08-19T01:21:44.245 INFO:teuthology.orchestra.run.smithi123.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-08-19T01:21:44.270 INFO:teuthology.orchestra.run.smithi123.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 714 kB/s | 18 kB 00:00 2024-08-19T01:21:44.296 INFO:teuthology.orchestra.run.smithi123.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 934 kB/s | 23 kB 00:00 2024-08-19T01:21:44.300 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-08-19T01:21:44.300 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-08-19T01:21:44.354 INFO:teuthology.orchestra.run.smithi123.stdout:(126/150): libarrow-9.0.0-11.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-08-19T01:21:44.379 INFO:teuthology.orchestra.run.smithi123.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 233 kB/s | 19 kB 00:00 2024-08-19T01:21:44.405 INFO:teuthology.orchestra.run.smithi123.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 797 kB/s | 20 kB 00:00 2024-08-19T01:21:44.430 INFO:teuthology.orchestra.run.smithi123.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-08-19T01:21:44.455 INFO:teuthology.orchestra.run.smithi123.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-08-19T01:21:44.480 INFO:teuthology.orchestra.run.smithi123.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-08-19T01:21:44.514 INFO:teuthology.orchestra.run.smithi123.stdout:(132/150): python3-pecan-1.4.2-3.el9.noarch.rpm 8.1 MB/s | 272 kB 00:00 2024-08-19T01:21:44.540 INFO:teuthology.orchestra.run.smithi123.stdout:(133/150): python3-portend-3.1.0-2.el9.noarch.r 656 kB/s | 16 kB 00:00 2024-08-19T01:21:44.565 INFO:teuthology.orchestra.run.smithi123.stdout:(134/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-08-19T01:21:44.590 INFO:teuthology.orchestra.run.smithi123.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-08-19T01:21:44.624 INFO:teuthology.orchestra.run.smithi123.stdout:(136/150): python3-kubernetes-26.1.0-2.el9.noar 4.2 MB/s | 1.0 MB 00:00 2024-08-19T01:21:44.649 INFO:teuthology.orchestra.run.smithi123.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 525 kB/s | 31 kB 00:00 2024-08-19T01:21:44.690 INFO:teuthology.orchestra.run.smithi123.stdout:(138/150): python3-grpcio-1.46.7-10.el9.x86_64. 3.8 MB/s | 2.0 MB 00:00 2024-08-19T01:21:44.716 INFO:teuthology.orchestra.run.smithi123.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-08-19T01:21:44.750 INFO:teuthology.orchestra.run.smithi123.stdout:(140/150): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-08-19T01:21:44.775 INFO:teuthology.orchestra.run.smithi123.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 613 kB/s | 36 kB 00:00 2024-08-19T01:21:44.808 INFO:teuthology.orchestra.run.smithi123.stdout:(142/150): python3-typing-extensions-4.12.2-2.e 811 kB/s | 74 kB 00:00 2024-08-19T01:21:44.834 INFO:teuthology.orchestra.run.smithi123.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-08-19T01:21:44.859 INFO:teuthology.orchestra.run.smithi123.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-08-19T01:21:44.901 INFO:teuthology.orchestra.run.smithi123.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-08-19T01:21:44.926 INFO:teuthology.orchestra.run.smithi123.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-08-19T01:21:44.960 INFO:teuthology.orchestra.run.smithi123.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-08-19T01:21:45.010 INFO:teuthology.orchestra.run.smithi123.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-19T01:21:45.185 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-08-19T01:21:45.202 INFO:teuthology.orchestra.run.smithi123.stdout:(149/150): librados2-19.1.0-1382.gde09dd01.el9. 12 MB/s | 3.4 MB 00:00 2024-08-19T01:21:45.268 INFO:teuthology.orchestra.run.smithi123.stdout:(150/150): librbd1-19.1.0-1382.gde09dd01.el9.x8 10 MB/s | 3.2 MB 00:00 2024-08-19T01:21:45.274 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-08-19T01:21:45.274 INFO:teuthology.orchestra.run.smithi123.stdout:Total 26 MB/s | 218 MB 00:08 2024-08-19T01:21:45.313 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-08-19T01:21:45.313 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-08-19T01:21:46.112 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-08-19T01:21:46.113 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-08-19T01:21:46.339 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-08-19T01:21:46.465 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-08-19T01:21:46.465 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-08-19T01:21:47.030 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-08-19T01:21:47.030 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-08-19T01:21:48.176 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-08-19T01:21:48.177 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-08-19T01:21:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-08-19T01:21:52.253 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-19T01:21:52.419 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-19T01:21:52.464 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-19T01:21:52.812 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-19T01:21:52.935 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 5/152 2024-08-19T01:21:53.061 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 5/152 2024-08-19T01:21:53.154 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:21:53.250 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:21:53.253 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-08-19T01:21:53.296 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 7/152 2024-08-19T01:21:53.323 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-19T01:21:53.325 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-19T01:21:53.348 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-19T01:21:53.350 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-19T01:21:53.392 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-19T01:21:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-08-19T01:21:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-08-19T01:21:53.592 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:21:53.681 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:21:53.710 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 13/152 2024-08-19T01:21:53.730 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-19T01:21:53.772 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 13/152 2024-08-19T01:21:53.794 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-19T01:21:53.817 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-19T01:21:53.841 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 5/152 2024-08-19T01:21:53.964 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 5/152 2024-08-19T01:21:53.993 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:21:54.005 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-08-19T01:21:54.087 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:21:54.087 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-19T01:21:54.113 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-19T01:21:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-19T01:21:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 7/152 2024-08-19T01:21:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-19T01:21:54.155 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-19T01:21:54.155 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-19T01:21:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-19T01:21:54.266 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-08-19T01:21:54.334 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-08-19T01:21:54.373 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-19T01:21:54.386 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-08-19T01:21:54.401 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-08-19T01:21:54.414 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:21:54.437 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-08-19T01:21:54.493 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-19T01:21:54.522 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-08-19T01:21:54.527 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:21:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 13/152 2024-08-19T01:21:54.581 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-08-19T01:21:54.614 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 5/152 2024-08-19T01:21:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 13/152 2024-08-19T01:21:54.627 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 22/152 2024-08-19T01:21:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-19T01:21:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-19T01:21:54.687 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/152 2024-08-19T01:21:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 5/152 2024-08-19T01:21:54.728 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/152 2024-08-19T01:21:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:21:54.759 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/152 2024-08-19T01:21:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:21:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 7/152 2024-08-19T01:21:54.875 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/152 2024-08-19T01:21:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-19T01:21:54.928 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-19T01:21:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 27/152 2024-08-19T01:21:54.969 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-19T01:21:54.987 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/152 2024-08-19T01:21:55.008 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-19T01:21:55.040 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/152 2024-08-19T01:21:55.078 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-08-19T01:21:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-08-19T01:21:55.117 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/152 2024-08-19T01:21:55.133 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-08-19T01:21:55.147 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9. 31/152 2024-08-19T01:21:55.161 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:21:55.193 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el 32/152 2024-08-19T01:21:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-19T01:21:55.219 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_6 33/152 2024-08-19T01:21:55.249 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-08-19T01:21:55.265 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:21:55.288 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 13/152 2024-08-19T01:21:55.293 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-08-19T01:21:55.295 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/152 2024-08-19T01:21:55.343 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-5.el9.noarch 35/152 2024-08-19T01:21:55.350 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 13/152 2024-08-19T01:21:55.367 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-08-19T01:21:55.372 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-19T01:21:55.391 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-19T01:21:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/152 2024-08-19T01:21:55.422 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-08-19T01:21:55.459 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 22/152 2024-08-19T01:21:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/152 2024-08-19T01:21:55.535 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/152 2024-08-19T01:21:55.563 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/152 2024-08-19T01:21:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/152 2024-08-19T01:21:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/152 2024-08-19T01:21:55.635 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/152 2024-08-19T01:21:55.686 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 27/152 2024-08-19T01:21:55.690 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-19T01:21:55.716 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/152 2024-08-19T01:21:55.738 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-19T01:21:55.760 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/152 2024-08-19T01:21:55.839 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/152 2024-08-19T01:21:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-08-19T01:21:55.878 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9. 31/152 2024-08-19T01:21:55.924 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el 32/152 2024-08-19T01:21:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_6 33/152 2024-08-19T01:21:55.951 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-19T01:21:55.987 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-08-19T01:21:56.022 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-08-19T01:21:56.023 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/152 2024-08-19T01:21:56.065 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-5.el9.noarch 35/152 2024-08-19T01:21:56.095 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-08-19T01:21:56.140 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-08-19T01:21:56.142 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/152 2024-08-19T01:21:56.180 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-packaging-20.9-5.el9.noarch 22/152 2024-08-19T01:21:56.185 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/152 2024-08-19T01:21:56.224 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/152 2024-08-19T01:21:56.257 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/152 2024-08-19T01:21:56.269 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/152 2024-08-19T01:21:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/152 2024-08-19T01:21:56.284 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/152 2024-08-19T01:21:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 41/152 2024-08-19T01:21:56.336 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/152 2024-08-19T01:21:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/152 2024-08-19T01:21:56.354 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/152 2024-08-19T01:21:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_6 43/152 2024-08-19T01:21:56.406 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 27/152 2024-08-19T01:21:56.425 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/152 2024-08-19T01:21:56.437 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/152 2024-08-19T01:21:56.480 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/152 2024-08-19T01:21:56.482 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 45/152 2024-08-19T01:21:56.556 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-19T01:21:56.558 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/152 2024-08-19T01:21:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9. 31/152 2024-08-19T01:21:56.594 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 47/152 2024-08-19T01:21:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el 32/152 2024-08-19T01:21:56.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-19T01:21:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_6 33/152 2024-08-19T01:21:56.702 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-19T01:21:56.734 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/152 2024-08-19T01:21:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-19T01:21:56.772 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-19T01:21:56.778 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : luarocks-3.9.2-5.el9.noarch 35/152 2024-08-19T01:21:56.814 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-19T01:21:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/152 2024-08-19T01:21:56.870 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/152 2024-08-19T01:21:56.947 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/152 2024-08-19T01:21:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-19T01:21:56.975 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 41/152 2024-08-19T01:21:56.986 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/152 2024-08-19T01:21:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-19T01:21:57.018 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/152 2024-08-19T01:21:57.041 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-19T01:21:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/152 2024-08-19T01:21:57.057 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_6 43/152 2024-08-19T01:21:57.089 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/152 2024-08-19T01:21:57.144 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 45/152 2024-08-19T01:21:57.149 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-19T01:21:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 57/152 2024-08-19T01:21:57.219 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-19T01:21:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-19T01:21:57.258 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 47/152 2024-08-19T01:21:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-19T01:21:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-19T01:21:57.294 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-19T01:21:57.341 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-19T01:21:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-19T01:21:57.376 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-19T01:21:57.419 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-19T01:21:57.461 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-19T01:21:57.506 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-19T01:21:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-19T01:21:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-19T01:21:57.567 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/152 2024-08-19T01:21:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-19T01:21:57.602 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-19T01:21:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-19T01:21:57.646 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-19T01:21:57.650 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/152 2024-08-19T01:21:57.679 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 41/152 2024-08-19T01:21:57.687 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-19T01:21:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/152 2024-08-19T01:21:57.760 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_6 43/152 2024-08-19T01:21:57.774 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-19T01:21:57.793 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-19T01:21:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/152 2024-08-19T01:21:57.813 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-19T01:21:57.837 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 57/152 2024-08-19T01:21:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-19T01:21:57.848 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 45/152 2024-08-19T01:21:57.869 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-19T01:21:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-19T01:21:57.903 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-19T01:21:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-19T01:21:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-19T01:21:57.928 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-19T01:21:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 47/152 2024-08-19T01:21:57.953 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-19T01:21:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-19T01:21:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-19T01:21:58.003 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-19T01:21:58.035 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-19T01:21:58.071 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-19T01:21:58.078 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-19T01:21:58.114 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-19T01:21:58.126 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-19T01:21:58.131 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-19T01:21:58.156 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-19T01:21:58.156 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-19T01:21:58.168 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-19T01:21:58.196 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-19T01:21:58.231 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-19T01:21:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-19T01:21:58.275 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-19T01:21:58.294 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-19T01:21:58.312 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-19T01:21:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-19T01:21:58.341 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-19T01:21:58.344 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-19T01:21:58.374 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-19T01:21:58.383 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-19T01:21:58.408 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-19T01:21:58.442 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-19T01:21:58.483 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-19T01:21:58.484 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-19T01:21:58.524 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 57/152 2024-08-19T01:21:58.564 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-19T01:21:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-19T01:21:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-19T01:21:58.689 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-19T01:21:58.727 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-19T01:21:58.818 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-19T01:21:58.823 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-19T01:21:58.863 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-19T01:21:58.866 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-19T01:21:58.891 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-19T01:21:58.896 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-19T01:21:58.925 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-19T01:21:58.969 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-19T01:21:59.002 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-19T01:21:59.007 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-19T01:21:59.038 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-19T01:21:59.056 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-19T01:21:59.068 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-19T01:21:59.103 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-19T01:21:59.119 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-19T01:21:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-19T01:21:59.164 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-19T01:21:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-19T01:21:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-19T01:21:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-19T01:21:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-19T01:21:59.312 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-19T01:21:59.361 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 83/152 2024-08-19T01:21:59.416 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-19T01:21:59.506 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-19T01:21:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-19T01:21:59.542 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-19T01:21:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-19T01:21:59.677 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-19T01:21:59.724 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-19T01:21:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-08-19T01:21:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-08-19T01:21:59.800 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-19T01:21:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-19T01:21:59.844 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-19T01:21:59.878 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-19T01:21:59.979 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-19T01:22:00.025 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 83/152 2024-08-19T01:22:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-19T01:22:00.186 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-19T01:22:00.232 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-19T01:22:00.407 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-08-19T01:22:00.437 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-08-19T01:22:00.454 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-19T01:22:00.496 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-19T01:22:00.515 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-19T01:22:00.530 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-19T01:22:00.532 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-19T01:22:00.629 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-19T01:22:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-19T01:22:00.677 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 83/152 2024-08-19T01:22:00.723 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-19T01:22:00.810 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-19T01:22:00.867 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-19T01:22:01.043 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-08-19T01:22:01.073 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-08-19T01:22:01.150 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-19T01:22:01.192 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-19T01:22:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-19T01:22:01.793 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-19T01:22:01.904 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-19T01:22:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-19T01:22:02.043 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-19T01:22:02.063 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-19T01:22:02.087 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-08-19T01:22:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-19T01:22:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 97/152 2024-08-19T01:22:02.554 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 97/152 2024-08-19T01:22:02.567 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-19T01:22:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 98/152 2024-08-19T01:22:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-19T01:22:02.658 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-19T01:22:02.686 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-19T01:22:02.719 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-08-19T01:22:03.028 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-19T01:22:03.074 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 97/152 2024-08-19T01:22:03.118 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-19T01:22:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-19T01:22:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 101/152 2024-08-19T01:22:03.177 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 97/152 2024-08-19T01:22:03.203 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 98/152 2024-08-19T01:22:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 101/152 2024-08-19T01:22:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-19T01:22:03.239 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-19T01:22:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 102/152 2024-08-19T01:22:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-19T01:22:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-08-19T01:22:03.594 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-19T01:22:03.633 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 97/152 2024-08-19T01:22:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-19T01:22:03.727 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 97/152 2024-08-19T01:22:03.754 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 98/152 2024-08-19T01:22:03.760 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 101/152 2024-08-19T01:22:03.783 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-19T01:22:03.825 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 101/152 2024-08-19T01:22:03.853 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 102/152 2024-08-19T01:22:04.263 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-19T01:22:04.294 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 101/152 2024-08-19T01:22:04.351 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 101/152 2024-08-19T01:22:04.379 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 102/152 2024-08-19T01:22:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:05.692 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:06.267 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 104/152 2024-08-19T01:22:06.313 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:06.358 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 104/152 2024-08-19T01:22:06.385 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T01:22:06.385 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T01:22:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 105/152 2024-08-19T01:22:06.683 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:06.726 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:06.907 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:06.946 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 104/152 2024-08-19T01:22:07.036 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 104/152 2024-08-19T01:22:07.036 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T01:22:07.037 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T01:22:07.037 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:07.072 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 105/152 2024-08-19T01:22:07.275 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 103/152 2024-08-19T01:22:07.313 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 104/152 2024-08-19T01:22:07.402 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 104/152 2024-08-19T01:22:07.402 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T01:22:07.402 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T01:22:07.402 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:07.431 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 105/152 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 105/152 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-08-19T01:22:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-08-19T01:22:18.418 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 105/152 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-08-19T01:22:18.457 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-08-19T01:22:18.458 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 106/152 2024-08-19T01:22:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 106/152 2024-08-19T01:22:18.615 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:18.615 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T01:22:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:18.634 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 106/152 2024-08-19T01:22:18.654 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 106/152 2024-08-19T01:22:18.654 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:18.654 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T01:22:18.654 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 105/152 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /root 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-08-19T01:22:18.880 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 106/152 2024-08-19T01:22:19.074 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 106/152 2024-08-19T01:22:19.075 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:19.075 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T01:22:19.075 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:20.137 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 107/152 2024-08-19T01:22:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 107/152 2024-08-19T01:22:20.164 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 107/152 2024-08-19T01:22:20.164 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.164 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T01:22:20.164 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T01:22:20.164 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T01:22:20.164 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 107/152 2024-08-19T01:22:20.179 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.179 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T01:22:20.179 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T01:22:20.180 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T01:22:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:20.387 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 108/152 2024-08-19T01:22:20.411 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 108/152 2024-08-19T01:22:20.411 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.411 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T01:22:20.411 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T01:22:20.411 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T01:22:20.411 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 108/152 2024-08-19T01:22:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 108/152 2024-08-19T01:22:20.443 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.443 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T01:22:20.443 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T01:22:20.443 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T01:22:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 107/152 2024-08-19T01:22:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 107/152 2024-08-19T01:22:20.579 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.579 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T01:22:20.579 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T01:22:20.579 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T01:22:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:20.804 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 108/152 2024-08-19T01:22:20.826 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 108/152 2024-08-19T01:22:20.826 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.827 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T01:22:20.827 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T01:22:20.827 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T01:22:20.827 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:20.846 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 109/152 2024-08-19T01:22:20.870 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 109/152 2024-08-19T01:22:20.870 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.870 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T01:22:20.870 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T01:22:20.870 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T01:22:20.870 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 109/152 2024-08-19T01:22:20.927 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 109/152 2024-08-19T01:22:20.927 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:20.927 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T01:22:20.927 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T01:22:20.927 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T01:22:20.927 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-19T01:22:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-19T01:22:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-19T01:22:21.093 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-19T01:22:21.097 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-19T01:22:21.156 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-19T01:22:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 109/152 2024-08-19T01:22:21.290 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-19T01:22:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 109/152 2024-08-19T01:22:21.295 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:21.295 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T01:22:21.295 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T01:22:21.295 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T01:22:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:21.339 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-19T01:22:21.357 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-19T01:22:21.405 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-19T01:22:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-19T01:22:21.417 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-19T01:22:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-19T01:22:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-19T01:22:21.505 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-19T01:22:21.521 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-19T01:22:21.574 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-19T01:22:21.644 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-19T01:22:21.681 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-19T01:22:21.705 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-19T01:22:21.713 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-19T01:22:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-19T01:22:21.740 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-19T01:22:21.755 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-19T01:22:21.755 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-19T01:22:21.786 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-19T01:22:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-19T01:22:21.833 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-19T01:22:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-19T01:22:22.051 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-19T01:22:22.097 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-19T01:22:22.129 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-19T01:22:22.157 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-19T01:22:22.204 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 121/152 2024-08-19T01:22:22.234 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 121/152 2024-08-19T01:22:22.262 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.e 122/152 2024-08-19T01:22:22.284 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 121/152 2024-08-19T01:22:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 121/152 2024-08-19T01:22:22.325 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01. 123/152 2024-08-19T01:22:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.e 122/152 2024-08-19T01:22:22.362 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 124/152 2024-08-19T01:22:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01. 123/152 2024-08-19T01:22:22.436 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 124/152 2024-08-19T01:22:22.618 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 121/152 2024-08-19T01:22:22.647 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 121/152 2024-08-19T01:22:22.678 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.e 122/152 2024-08-19T01:22:22.740 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01. 123/152 2024-08-19T01:22:22.778 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 124/152 2024-08-19T01:22:23.403 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 124/152 2024-08-19T01:22:23.439 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 125/152 2024-08-19T01:22:23.540 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 124/152 2024-08-19T01:22:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 125/152 2024-08-19T01:22:23.814 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 124/152 2024-08-19T01:22:23.846 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 125/152 2024-08-19T01:22:24.402 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 125/152 2024-08-19T01:22:24.431 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 126/152 2024-08-19T01:22:24.532 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 126/152 2024-08-19T01:22:24.594 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 125/152 2024-08-19T01:22:24.621 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 126/152 2024-08-19T01:22:24.694 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el 127/152 2024-08-19T01:22:24.727 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 126/152 2024-08-19T01:22:24.733 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 128/152 2024-08-19T01:22:24.757 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 128/152 2024-08-19T01:22:24.757 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:24.758 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T01:22:24.758 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T01:22:24.758 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T01:22:24.758 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:24.803 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 125/152 2024-08-19T01:22:24.809 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 129/152 2024-08-19T01:22:24.820 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 129/152 2024-08-19T01:22:24.830 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 126/152 2024-08-19T01:22:24.891 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el 127/152 2024-08-19T01:22:24.918 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 130/152 2024-08-19T01:22:24.923 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 128/152 2024-08-19T01:22:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 126/152 2024-08-19T01:22:24.946 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 128/152 2024-08-19T01:22:24.946 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:24.946 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T01:22:24.946 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T01:22:24.947 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T01:22:24.947 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:24.973 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 131/152 2024-08-19T01:22:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 129/152 2024-08-19T01:22:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 129/152 2024-08-19T01:22:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el 127/152 2024-08-19T01:22:25.103 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 130/152 2024-08-19T01:22:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 128/152 2024-08-19T01:22:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 128/152 2024-08-19T01:22:25.155 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:25.155 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T01:22:25.155 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T01:22:25.155 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T01:22:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:25.156 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 131/152 2024-08-19T01:22:25.199 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 129/152 2024-08-19T01:22:25.210 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 129/152 2024-08-19T01:22:25.308 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 130/152 2024-08-19T01:22:25.355 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 131/152 2024-08-19T01:22:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-19T01:22:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 133/152 2024-08-19T01:22:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 133/152 2024-08-19T01:22:25.935 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:25.935 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T01:22:25.935 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T01:22:25.935 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T01:22:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1382.gde09d 134/152 2024-08-19T01:22:26.002 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1382.gde09d 134/152 2024-08-19T01:22:26.002 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.002 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T01:22:26.002 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:26.054 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 135/152 2024-08-19T01:22:26.078 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 135/152 2024-08-19T01:22:26.078 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.078 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-19T01:22:26.078 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-19T01:22:26.078 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-19T01:22:26.078 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:26.121 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-19T01:22:26.151 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 133/152 2024-08-19T01:22:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 133/152 2024-08-19T01:22:26.174 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.175 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T01:22:26.175 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T01:22:26.175 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T01:22:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:26.218 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1382.gde09d 134/152 2024-08-19T01:22:26.240 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1382.gde09d 134/152 2024-08-19T01:22:26.240 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.240 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T01:22:26.240 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:26.268 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-19T01:22:26.286 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 135/152 2024-08-19T01:22:26.302 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 133/152 2024-08-19T01:22:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 135/152 2024-08-19T01:22:26.309 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.309 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-19T01:22:26.310 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-19T01:22:26.310 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-19T01:22:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:26.325 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 133/152 2024-08-19T01:22:26.326 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.326 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T01:22:26.326 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T01:22:26.326 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T01:22:26.326 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:26.369 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1382.gde09d 134/152 2024-08-19T01:22:26.380 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 136/152 2024-08-19T01:22:26.391 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1382.gde09d 134/152 2024-08-19T01:22:26.392 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.392 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T01:22:26.392 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:26.404 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 136/152 2024-08-19T01:22:26.404 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.404 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T01:22:26.404 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T01:22:26.404 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T01:22:26.404 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:26.436 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 135/152 2024-08-19T01:22:26.459 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 135/152 2024-08-19T01:22:26.460 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.460 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-19T01:22:26.460 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-19T01:22:26.460 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-19T01:22:26.460 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:26.628 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 136/152 2024-08-19T01:22:26.651 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 136/152 2024-08-19T01:22:26.651 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.651 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T01:22:26.651 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T01:22:26.651 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T01:22:26.651 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:26.771 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 136/152 2024-08-19T01:22:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 136/152 2024-08-19T01:22:26.794 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T01:22:26.794 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T01:22:26.794 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T01:22:26.794 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T01:22:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 137/152 2024-08-19T01:22:30.983 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 138/152 2024-08-19T01:22:31.026 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 139/152 2024-08-19T01:22:31.056 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 140/152 2024-08-19T01:22:31.083 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-19T01:22:31.265 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 142/152 2024-08-19T01:22:31.304 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-08-19T01:22:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 137/152 2024-08-19T01:22:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 138/152 2024-08-19T01:22:31.359 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_ 144/152 2024-08-19T01:22:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 139/152 2024-08-19T01:22:31.421 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 140/152 2024-08-19T01:22:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-19T01:22:31.548 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 137/152 2024-08-19T01:22:31.590 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 138/152 2024-08-19T01:22:31.626 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 142/152 2024-08-19T01:22:31.630 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 139/152 2024-08-19T01:22:31.662 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 140/152 2024-08-19T01:22:31.678 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-08-19T01:22:31.689 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-19T01:22:31.708 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_ 144/152 2024-08-19T01:22:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 142/152 2024-08-19T01:22:31.929 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-08-19T01:22:31.965 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:31.973 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_ 144/152 2024-08-19T01:22:31.985 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:32.312 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:32.355 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:32.632 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:32.671 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:32.690 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-08-19T01:22:32.827 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 146/152 2024-08-19T01:22:32.920 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 147/152 2024-08-19T01:22:32.961 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-19T01:22:33.045 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-19T01:22:33.074 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-19T01:22:33.074 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-19T01:22:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-19T01:22:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 146/152 2024-08-19T01:22:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 147/152 2024-08-19T01:22:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-19T01:22:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-19T01:22:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-19T01:22:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-19T01:22:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-19T01:22:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 146/152 2024-08-19T01:22:33.650 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 147/152 2024-08-19T01:22:33.692 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-19T01:22:33.777 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-19T01:22:33.838 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-19T01:22:33.838 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-19T01:22:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-19T01:22:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 1/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 2/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 3/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 4/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1382.gde09d 5/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:22:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 7/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 8/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 9/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 10/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 11/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 13/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 14/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_ 15/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 16/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 17/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_6 18/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 19/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 20/152 2024-08-19T01:22:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el 21/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9. 22/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_6 23/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 24/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 25/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 26/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 27/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 28/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 29/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01. 30/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 31/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 32/152 2024-08-19T01:22:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 33/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el 34/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 35/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.e 36/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 37/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 38/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 39/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-19T01:22:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-19T01:22:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-08-19T01:22:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 77/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-19T01:22:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-19T01:22:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-19T01:22:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-19T01:22:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 122/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-19T01:22:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-19T01:22:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 149/152 2024-08-19T01:22:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-19T01:22:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 151/152 2024-08-19T01:22:39.283 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:39.283 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 1/152 2024-08-19T01:22:39.283 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 2/152 2024-08-19T01:22:39.283 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 3/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 4/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1382.gde09d 5/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 7/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 8/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 9/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 10/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 11/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 13/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 14/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_ 15/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 16/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 17/152 2024-08-19T01:22:39.284 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_6 18/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 19/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 20/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el 21/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9. 22/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_6 23/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 24/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 25/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 26/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 27/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 28/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 29/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01. 30/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 31/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 32/152 2024-08-19T01:22:39.285 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 33/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el 34/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 35/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.e 36/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 37/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 38/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 39/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-19T01:22:39.286 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-19T01:22:39.287 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-19T01:22:39.288 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 77/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-19T01:22:39.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/152 2024-08-19T01:22:39.290 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-19T01:22:39.291 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 122/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-19T01:22:39.292 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-19T01:22:39.293 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-08-19T01:22:39.294 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-19T01:22:39.294 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 149/152 2024-08-19T01:22:39.294 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-19T01:22:39.294 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 151/152 2024-08-19T01:22:39.511 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:39.511 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 1/152 2024-08-19T01:22:39.511 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 2/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 3/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 4/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1382.gde09d 5/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 6/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 7/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 8/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 9/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 10/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 11/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 12/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 13/152 2024-08-19T01:22:39.512 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 14/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_ 15/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 16/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 17/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_6 18/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86 19/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 20/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el 21/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.1.0-1382.gde09dd01.el9. 22/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_6 23/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 24/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 25/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 26/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 27/152 2024-08-19T01:22:39.513 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 28/152 2024-08-19T01:22:39.514 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 29/152 2024-08-19T01:22:39.514 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01. 30/152 2024-08-19T01:22:39.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noa 31/152 2024-08-19T01:22:39.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.n 32/152 2024-08-19T01:22:39.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1382.gde0 33/152 2024-08-19T01:22:39.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el 34/152 2024-08-19T01:22:39.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 35/152 2024-08-19T01:22:39.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.e 36/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 37/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 38/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 39/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-19T01:22:39.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-08-19T01:22:39.526 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-19T01:22:39.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 77/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-19T01:22:39.528 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-19T01:22:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-19T01:22:39.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 122/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-19T01:22:39.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-19T01:22:39.532 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-19T01:22:39.533 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-08-19T01:22:39.533 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-19T01:22:39.533 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 149/152 2024-08-19T01:22:39.533 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-19T01:22:39.533 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 151/152 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.2-8.el9.noarch 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.7.4-5.el9.x86_64 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T01:22:41.107 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.108 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.4-9.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-19T01:22:41.109 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.110 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T01:22:41.111 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-13.14-1.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server-13.14-1.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T01:22:41.112 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T01:22:41.113 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T01:22:41.114 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T01:22:41.115 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T01:22:41.116 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:22:41.117 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-08-19T01:22:41.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:41.444 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.2-8.el9.noarch 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.7.4-5.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.445 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.446 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.4-9.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-19T01:22:41.447 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.448 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-19T01:22:41.449 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-13.14-1.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server-13.14-1.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.450 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T01:22:41.451 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T01:22:41.452 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T01:22:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T01:22:41.454 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T01:22:41.455 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T01:22:41.456 INFO:teuthology.orchestra.run.smithi123.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-19T01:22:41.456 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-08-19T01:22:41.456 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T01:22:41.456 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-19T01:22:41.456 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:22:41.457 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-08-19T01:22:41.459 DEBUG:teuthology.parallel:result is None 2024-08-19T01:22:41.771 DEBUG:teuthology.parallel:result is None 2024-08-19T01:22:41.774 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-19T01:22:41.774 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: automake-1.16.2-8.el9.noarch 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: bison-3.7.4-5.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.775 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.776 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top-2:19.1.0-1382.gde09dd01.el9.noarch 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: flex-2.6.4-9.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.777 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-19T01:22:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-13.14-1.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server-13.14-1.el9.x86_64 2024-08-19T01:22:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T01:22:41.780 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T01:22:41.781 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T01:22:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T01:22:41.783 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.784 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.1.0-1382.gde09dd01.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:22:41.785 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-08-19T01:22:42.092 DEBUG:teuthology.parallel:result is None 2024-08-19T01:22:42.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:22:42.240 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T01:22:42.264 INFO:teuthology.orchestra.run.smithi064.stdout:19.1.0-1382.gde09dd01.el9 2024-08-19T01:22:42.264 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1382.gde09dd01.el9 2024-08-19T01:22:42.264 INFO:teuthology.task.install:The correct ceph version 19.1.0-1382.gde09dd01 is installed. 2024-08-19T01:22:42.266 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:22:42.411 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T01:22:42.434 INFO:teuthology.orchestra.run.smithi123.stdout:19.1.0-1382.gde09dd01.el9 2024-08-19T01:22:42.434 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1382.gde09dd01.el9 2024-08-19T01:22:42.434 INFO:teuthology.task.install:The correct ceph version 19.1.0-1382.gde09dd01 is installed. 2024-08-19T01:22:42.436 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:22:42.582 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T01:22:42.605 INFO:teuthology.orchestra.run.smithi161.stdout:19.1.0-1382.gde09dd01.el9 2024-08-19T01:22:42.605 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1382.gde09dd01.el9 2024-08-19T01:22:42.605 INFO:teuthology.task.install:The correct ceph version 19.1.0-1382.gde09dd01 is installed. 2024-08-19T01:22:42.607 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-19T01:22:42.607 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:22:42.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T01:22:42.634 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:22:42.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T01:22:42.662 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:22:42.662 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T01:22:42.690 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-19T01:22:42.690 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:22:42.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T01:22:42.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T01:22:42.780 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:22:42.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T01:22:42.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T01:22:42.872 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:22:42.872 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T01:22:42.899 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T01:22:42.966 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-19T01:22:42.966 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:22:42.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T01:22:42.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T01:22:43.059 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:22:43.059 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T01:22:43.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T01:22:43.147 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:22:43.148 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T01:22:43.172 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T01:22:43.238 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-19T01:22:43.239 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:22:43.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T01:22:43.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T01:22:43.330 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:22:43.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T01:22:43.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T01:22:43.421 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:22:43.421 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T01:22:43.446 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T01:22:43.510 INFO:teuthology.run_tasks:Running task cephadm... 2024-08-19T01:22:43.596 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-08-19T01:22:43.597 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:22:43.597 INFO:tasks.cephadm:Cluster fsid is 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:22:43.597 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-08-19T01:22:43.597 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.64', 'mon.b': '172.21.15.123', 'mon.c': '172.21.15.161'} 2024-08-19T01:22:43.597 INFO:tasks.cephadm:First mon is mon.a on smithi064 2024-08-19T01:22:43.598 INFO:tasks.cephadm:First mgr is x 2024-08-19T01:22:43.598 INFO:tasks.cephadm:Normalizing hostnames... 2024-08-19T01:22:43.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-08-19T01:22:43.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-08-19T01:22:43.648 DEBUG:teuthology.orchestra.run.smithi161:> sudo hostname $(hostname -s) 2024-08-19T01:22:43.673 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-08-19T01:22:43.673 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:22:43.815 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55869/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.0-1382-gde09dd01', 'node_name': '172.21.5.40+adami10', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.0-1382.gde09dd01'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-08-18 16:22:46.893995', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-08-19T01:22:43.958 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 de09dd018de4baddfa6ca4d43bc8fb784cf75833 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:22:43.959 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm 2024-08-19T01:22:43.959 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm 2024-08-19T01:22:43.960 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:44.349 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 784608 Aug 19 01:22 /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:44.349 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:44.785 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-r--r--. 1 ubuntu ubuntu 784608 Aug 19 01:22 /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:44.786 DEBUG:teuthology.orchestra.run.smithi161:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:45.575 INFO:teuthology.orchestra.run.smithi161.stdout:-rw-r--r--. 1 ubuntu ubuntu 784608 Aug 19 01:22 /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:45.576 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:45.595 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:45.612 DEBUG:teuthology.orchestra.run.smithi161:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-19T01:22:45.647 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 on all hosts... 2024-08-19T01:22:45.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 pull 2024-08-19T01:22:45.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 pull 2024-08-19T01:22:45.655 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 pull 2024-08-19T01:22:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T01:22:47.268 INFO:teuthology.orchestra.run.smithi123.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T01:22:47.324 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T01:23:14.358 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:23:14.358 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc)", 2024-08-19T01:23:14.358 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba", 2024-08-19T01:23:14.358 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-08-19T01:23:14.358 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9" 2024-08-19T01:23:14.358 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:23:14.358 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:23:25.635 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-08-19T01:23:25.635 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc)", 2024-08-19T01:23:25.635 INFO:teuthology.orchestra.run.smithi123.stdout: "image_id": "ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba", 2024-08-19T01:23:25.635 INFO:teuthology.orchestra.run.smithi123.stdout: "repo_digests": [ 2024-08-19T01:23:25.635 INFO:teuthology.orchestra.run.smithi123.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9" 2024-08-19T01:23:25.635 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-08-19T01:23:25.635 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-08-19T01:23:28.775 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-08-19T01:23:28.775 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc)", 2024-08-19T01:23:28.775 INFO:teuthology.orchestra.run.smithi161.stdout: "image_id": "ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba", 2024-08-19T01:23:28.775 INFO:teuthology.orchestra.run.smithi161.stdout: "repo_digests": [ 2024-08-19T01:23:28.775 INFO:teuthology.orchestra.run.smithi161.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9" 2024-08-19T01:23:28.775 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-08-19T01:23:28.776 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-08-19T01:23:28.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-08-19T01:23:28.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-08-19T01:23:28.870 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph 2024-08-19T01:23:28.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-08-19T01:23:28.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-08-19T01:23:28.961 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /etc/ceph 2024-08-19T01:23:28.986 INFO:tasks.cephadm:Writing seed config... 2024-08-19T01:23:28.987 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-08-19T01:23:28.988 INFO:tasks.cephadm: override: [client] debug client = 20 2024-08-19T01:23:28.988 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-08-19T01:23:28.988 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-08-19T01:23:28.988 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-08-19T01:23:28.988 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-08-19T01:23:28.988 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-08-19T01:23:28.988 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-08-19T01:23:28.989 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-08-19T01:23:28.990 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-08-19T01:23:28.991 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-08-19T01:23:28.991 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-08-19T01:23:28.991 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-08-19T01:23:28.991 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-08-19T01:23:28.991 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-08-19T01:23:28.991 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-08-19T01:23:28.991 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-08-19T01:23:28.992 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-08-19T01:23:28.992 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-08-19T01:23:28.992 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-08-19T01:23:28.992 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-08-19T01:23:28.992 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-08-19T01:23:28.992 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-08-19T01:23:28.992 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-08-19T01:23:28.993 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-08-19T01:23:28.993 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-08-19T01:23:28.993 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-08-19T01:23:28.993 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-08-19T01:23:28.993 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-08-19T01:23:28.993 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-08-19T01:23:28.995 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:23:28.995 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-08-19T01:23:29.013 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 88c1a048-5dc9-11ef-bccf-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-08-19T01:23:29.013 DEBUG:teuthology.orchestra.run.smithi064:mon.a> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a.service 2024-08-19T01:23:29.055 DEBUG:teuthology.orchestra.run.smithi064:mgr.x> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.x.service 2024-08-19T01:23:29.100 INFO:tasks.cephadm:Bootstrapping... 2024-08-19T01:23:29.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 -v bootstrap --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.64 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:23:29.313 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-08-19T01:23:29.313 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833', '-v', 'bootstrap', '--fsid', '88c1a048-5dc9-11ef-bccf-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.64', '--skip-admin-label'] 2024-08-19T01:23:29.313 INFO:teuthology.orchestra.run.smithi064.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-08-19T01:23:29.314 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying podman|docker is present... 2024-08-19T01:23:29.342 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 5.1.2 2024-08-19T01:23:29.343 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying lvm2 is present... 2024-08-19T01:23:29.343 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying time synchronization is in place... 2024-08-19T01:23:29.348 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-19T01:23:29.348 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-19T01:23:29.354 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-19T01:23:29.355 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-08-19T01:23:29.361 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-08-19T01:23:29.368 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-08-19T01:23:29.369 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-08-19T01:23:29.369 INFO:teuthology.orchestra.run.smithi064.stdout:Repeating the final host check... 2024-08-19T01:23:29.400 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 5.1.2 2024-08-19T01:23:29.400 INFO:teuthology.orchestra.run.smithi064.stdout:podman (/bin/podman) version 5.1.2 is present 2024-08-19T01:23:29.400 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl is present 2024-08-19T01:23:29.401 INFO:teuthology.orchestra.run.smithi064.stdout:lvcreate is present 2024-08-19T01:23:29.406 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-19T01:23:29.406 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-19T01:23:29.412 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-19T01:23:29.412 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-08-19T01:23:29.419 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-08-19T01:23:29.428 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-08-19T01:23:29.428 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-08-19T01:23:29.429 INFO:teuthology.orchestra.run.smithi064.stdout:Host looks OK 2024-08-19T01:23:29.429 INFO:teuthology.orchestra.run.smithi064.stdout:Cluster fsid: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:29.429 INFO:teuthology.orchestra.run.smithi064.stdout:Acquiring lock 140002058721792 on /run/cephadm/88c1a048-5dc9-11ef-bccf-c7b262605968.lock 2024-08-19T01:23:29.429 INFO:teuthology.orchestra.run.smithi064.stdout:Lock 140002058721792 acquired on /run/cephadm/88c1a048-5dc9-11ef-bccf-c7b262605968.lock 2024-08-19T01:23:29.429 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 3300 ... 2024-08-19T01:23:29.429 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 6789 ... 2024-08-19T01:23:29.430 INFO:teuthology.orchestra.run.smithi064.stdout:Base mon IP(s) is [172.21.15.64:3300, 172.21.15.64:6789], mon addrv is [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-08-19T01:23:29.431 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.64 metric 100 2024-08-19T01:23:29.431 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.64 metric 100 2024-08-19T01:23:29.434 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-08-19T01:23:29.434 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-08-19T01:23:29.434 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-08-19T01:23:29.436 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-08-19T01:23:29.437 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-08-19T01:23:29.437 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-19T01:23:29.437 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-08-19T01:23:29.438 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8df/64 scope link noprefixroute 2024-08-19T01:23:29.438 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-19T01:23:29.439 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-08-19T01:23:29.439 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-08-19T01:23:29.439 INFO:teuthology.orchestra.run.smithi064.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-08-19T01:23:29.439 INFO:teuthology.orchestra.run.smithi064.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-08-19T01:23:29.439 INFO:teuthology.orchestra.run.smithi064.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T01:23:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba 2024-08-19T01:23:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T01:23:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Getting image source signatures 2024-08-19T01:23:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:7556061847b3cc957dc118ff2516cc1cd809c4272dcb16074c5c7fe28afc4774 2024-08-19T01:23:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-08-19T01:23:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying config sha256:ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba 2024-08-19T01:23:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Writing manifest to image destination 2024-08-19T01:23:35.635 INFO:teuthology.orchestra.run.smithi064.stdout:ceph: stdout ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc) 2024-08-19T01:23:35.635 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph version: ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc) 2024-08-19T01:23:35.635 INFO:teuthology.orchestra.run.smithi064.stdout:Extracting ceph user uid/gid from container image... 2024-08-19T01:23:41.770 INFO:teuthology.orchestra.run.smithi064.stdout:stat: stdout 167 167 2024-08-19T01:23:41.770 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial keys... 2024-08-19T01:23:43.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQCensJm1ZUeBRAAQTpe6Xf6QBjHAO0i/jr0qg== 2024-08-19T01:23:43.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQCfnsJmJ2rJERAAh22d+15Rd7XvjY7obnVCTw== 2024-08-19T01:23:44.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQCgnsJmOgkhCRAAuYBDznWLPOJzoi+O5EefXA== 2024-08-19T01:23:44.720 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial monmap... 2024-08-19T01:23:45.584 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-19T01:23:45.584 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-08-19T01:23:45.584 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:45.585 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-19T01:23:45.585 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool for a [v2:172.21.15.64:3300,v1:172.21.15.64:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-19T01:23:45.585 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-08-19T01:23:45.585 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: set fsid to 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:45.585 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-19T01:23:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:23:45.586 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mon... 2024-08-19T01:23:46.505 INFO:teuthology.orchestra.run.smithi064.stdout:create mon.a on 2024-08-19T01:23:46.715 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-08-19T01:23:47.131 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-08-19T01:23:47.642 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968.target. 2024-08-19T01:23:47.642 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968.target. 2024-08-19T01:23:47.984 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a 2024-08-19T01:23:47.984 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a.service: Unit ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a.service not loaded. 2024-08-19T01:23:48.150 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968.target.wants/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a.service → /etc/systemd/system/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@.service. 2024-08-19T01:23:48.775 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T01:23:48.775 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-08-19T01:23:48.782 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T01:23:48.782 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-08-19T01:23:48.782 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-08-19T01:23:48.782 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-08-19T01:23:48.783 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon to start... 2024-08-19T01:23:48.783 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon... 2024-08-19T01:23:49.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:49 smithi064 ceph-mon[18148]: from='client.? 172.21.15.64:0/1018070131' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-08-19T01:23:49.801 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout cluster: 2024-08-19T01:23:49.801 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout id: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:49.801 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout services: 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.496211s) 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout data: 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pgs: 2024-08-19T01:23:49.802 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.164+0000 7f71834fd640 1 Processor -- start 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.164+0000 7f71834fd640 1 -- start start 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.165+0000 7f71834fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c106720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.165+0000 7f71834fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f717c106cf0 con 0x7f717c106320 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.165+0000 7f7181299640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.165+0000 7f7181299640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42460/0 (socket says 172.21.15.64:42460) 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.165+0000 7f7181299640 1 -- 172.21.15.64:0/95987266 learned_addr learned my addr 172.21.15.64:0/95987266 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.165+0000 7f7181299640 1 -- 172.21.15.64:0/95987266 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717c107520 con 0x7f717c106320 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.167+0000 7f7181299640 1 --2- 172.21.15.64:0/95987266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c106720 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7178009820 tx=0x7f717802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841c033105bd5238 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.167+0000 7f7173fff640 1 -- 172.21.15.64:0/95987266 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7178037070 con 0x7f717c106320 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.167+0000 7f7173fff640 1 -- 172.21.15.64:0/95987266 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7178032440 con 0x7f717c106320 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.167+0000 7f7173fff640 1 -- 172.21.15.64:0/95987266 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7178032740 con 0x7f717c106320 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 -- 172.21.15.64:0/95987266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 msgr2=0x7f717c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 --2- 172.21.15.64:0/95987266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c106720 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7178009820 tx=0x7f717802a270 comp rx=0 tx=0).stop 2024-08-19T01:23:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 -- 172.21.15.64:0/95987266 shutdown_connections 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 --2- 172.21.15.64:0/95987266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c106720 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 -- 172.21.15.64:0/95987266 >> 172.21.15.64:0/95987266 conn(0x7f717c101b10 msgr2=0x7f717c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 -- 172.21.15.64:0/95987266 shutdown_connections 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 -- 172.21.15.64:0/95987266 wait complete. 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 Processor -- start 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.168+0000 7f71834fd640 1 -- start start 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f71834fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f71834fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f717c107ac0 con 0x7f717c106320 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f7181299640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f7181299640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c19ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42474/0 (socket says 172.21.15.64:42474) 2024-08-19T01:23:49.804 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f7181299640 1 -- 172.21.15.64:0/1018070131 learned_addr learned my addr 172.21.15.64:0/1018070131 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f7181299640 1 -- 172.21.15.64:0/1018070131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717c19b170 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f7181299640 1 --2- 172.21.15.64:0/1018070131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c19ac30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f71780090b0 tx=0x7f71780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f71727fc640 1 -- 172.21.15.64:0/1018070131 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7178041070 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f71727fc640 1 -- 172.21.15.64:0/1018070131 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717803c7a0 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f71727fc640 1 -- 172.21.15.64:0/1018070131 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7178037040 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f717c19b400 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.169+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f717c19e0f0 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.170+0000 7f71727fc640 1 -- 172.21.15.64:0/1018070131 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7178044070 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.170+0000 7f71727fc640 1 -- 172.21.15.64:0/1018070131 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f717803d3f0 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.170+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f717c108110 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.173+0000 7f71727fc640 1 -- 172.21.15.64:0/1018070131 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f7178046450 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.234+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f717c02d840 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.234+0000 7f71727fc640 1 -- 172.21.15.64:0/1018070131 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f7178046630 con 0x7f717c106320 2024-08-19T01:23:49.805 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.235+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 msgr2=0x7f717c19ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.235+0000 7f71834fd640 1 --2- 172.21.15.64:0/1018070131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c19ac30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f71780090b0 tx=0x7f71780046c0 comp rx=0 tx=0).stop 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.235+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 shutdown_connections 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.235+0000 7f71834fd640 1 --2- 172.21.15.64:0/1018070131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f717c106320 0x7f717c19ac30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.235+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 >> 172.21.15.64:0/1018070131 conn(0x7f717c101b10 msgr2=0x7f717c1918f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.235+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 shutdown_connections 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:49.235+0000 7f71834fd640 1 -- 172.21.15.64:0/1018070131 wait complete. 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:mon is available 2024-08-19T01:23:49.806 INFO:teuthology.orchestra.run.smithi064.stdout:Assimilating anything we can from ceph.conf... 2024-08-19T01:23:50.849 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:50 smithi064 ceph-mon[18148]: from='client.? 172.21.15.64:0/3658397907' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-19T01:23:50.849 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:50 smithi064 ceph-mon[18148]: from='client.? 172.21.15.64:0/3658397907' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-19T01:23:50.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.174+0000 7f1663245640 1 Processor -- start 2024-08-19T01:23:50.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.174+0000 7f1663245640 1 -- start start 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.174+0000 7f1663245640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c10a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.174+0000 7f1663245640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f165c07be70 con 0x7f165c107ef0 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.174+0000 7f1660fe1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c10a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.174+0000 7f1660fe1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c10a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42488/0 (socket says 172.21.15.64:42488) 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.174+0000 7f1660fe1640 1 -- 172.21.15.64:0/2836856514 learned_addr learned my addr 172.21.15.64:0/2836856514 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.175+0000 7f1660fe1640 1 -- 172.21.15.64:0/2836856514 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165c10a820 con 0x7f165c107ef0 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.175+0000 7f1660fe1640 1 --2- 172.21.15.64:0/2836856514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c10a2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1650009820 tx=0x7f165002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d3141dbe508741 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.175+0000 7f165b7fe640 1 -- 172.21.15.64:0/2836856514 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1650037070 con 0x7f165c107ef0 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.175+0000 7f165b7fe640 1 -- 172.21.15.64:0/2836856514 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1650032440 con 0x7f165c107ef0 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.175+0000 7f165b7fe640 1 -- 172.21.15.64:0/2836856514 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1650032740 con 0x7f165c107ef0 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.176+0000 7f1663245640 1 -- 172.21.15.64:0/2836856514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 msgr2=0x7f165c10a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.176+0000 7f1663245640 1 --2- 172.21.15.64:0/2836856514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c10a2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1650009820 tx=0x7f165002a270 comp rx=0 tx=0).stop 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.176+0000 7f1663245640 1 -- 172.21.15.64:0/2836856514 shutdown_connections 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.176+0000 7f1663245640 1 --2- 172.21.15.64:0/2836856514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c10a2e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.176+0000 7f1663245640 1 -- 172.21.15.64:0/2836856514 >> 172.21.15.64:0/2836856514 conn(0x7f165c101b90 msgr2=0x7f165c103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.176+0000 7f1663245640 1 -- 172.21.15.64:0/2836856514 shutdown_connections 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.176+0000 7f1663245640 1 -- 172.21.15.64:0/2836856514 wait complete. 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1663245640 1 Processor -- start 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1663245640 1 -- start start 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1663245640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c107770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1663245640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f165c10ad60 con 0x7f165c107ef0 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1660fe1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1660fe1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42498/0 (socket says 172.21.15.64:42498) 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1660fe1640 1 -- 172.21.15.64:0/3658397907 learned_addr learned my addr 172.21.15.64:0/3658397907 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.177+0000 7f1660fe1640 1 -- 172.21.15.64:0/3658397907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165c107cb0 con 0x7f165c107ef0 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1660fe1640 1 --2- 172.21.15.64:0/3658397907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c107770 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f16500090b0 tx=0x7f16500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1650041070 con 0x7f165c107ef0 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165003c7a0 con 0x7f165c107ef0 2024-08-19T01:23:50.861 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165c105ea0 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1650037040 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f165c1063e0 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1650044070 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.178+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f165003d3f0 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.179+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f165c108350 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.181+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f1650046450 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.240+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f165c1945f0 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.243+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f1650046630 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.244+0000 7f1659ffb640 1 -- 172.21.15.64:0/3658397907 <== mon.0 v2:172.21.15.64:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1650056070 con 0x7f165c107ef0 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.245+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 msgr2=0x7f165c107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.245+0000 7f1663245640 1 --2- 172.21.15.64:0/3658397907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c107770 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f16500090b0 tx=0x7f16500046c0 comp rx=0 tx=0).stop 2024-08-19T01:23:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.245+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 shutdown_connections 2024-08-19T01:23:50.863 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.245+0000 7f1663245640 1 --2- 172.21.15.64:0/3658397907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f165c107ef0 0x7f165c107770 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:50.863 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.245+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 >> 172.21.15.64:0/3658397907 conn(0x7f165c101b90 msgr2=0x7f165c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:50.863 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.245+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 shutdown_connections 2024-08-19T01:23:50.863 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:50.245+0000 7f1663245640 1 -- 172.21.15.64:0/3658397907 wait complete. 2024-08-19T01:23:50.863 INFO:teuthology.orchestra.run.smithi064.stdout:Generating new minimal ceph.conf... 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.246+0000 7fe182114640 1 Processor -- start 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.246+0000 7fe182114640 1 -- start start 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.246+0000 7fe182114640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c1078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.246+0000 7fe182114640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c107ea0 con 0x7fe17c1074d0 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.246+0000 7fe17b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c1078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.246+0000 7fe17b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c1078d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42506/0 (socket says 172.21.15.64:42506) 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.246+0000 7fe17b7fe640 1 -- 172.21.15.64:0/4184124758 learned_addr learned my addr 172.21.15.64:0/4184124758 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.247+0000 7fe17b7fe640 1 -- 172.21.15.64:0/4184124758 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c108680 con 0x7fe17c1074d0 2024-08-19T01:23:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.247+0000 7fe17b7fe640 1 --2- 172.21.15.64:0/4184124758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c1078d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009820 tx=0x7fe16c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b3b750aac277bc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.247+0000 7fe17a7fc640 1 -- 172.21.15.64:0/4184124758 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe16c037070 con 0x7fe17c1074d0 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.247+0000 7fe17a7fc640 1 -- 172.21.15.64:0/4184124758 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe16c032900 con 0x7fe17c1074d0 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.247+0000 7fe17a7fc640 1 -- 172.21.15.64:0/4184124758 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe16c030850 con 0x7fe17c1074d0 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.248+0000 7fe182114640 1 -- 172.21.15.64:0/4184124758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 msgr2=0x7fe17c1078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.248+0000 7fe182114640 1 --2- 172.21.15.64:0/4184124758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c1078d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009820 tx=0x7fe16c02a270 comp rx=0 tx=0).stop 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.248+0000 7fe182114640 1 -- 172.21.15.64:0/4184124758 shutdown_connections 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.248+0000 7fe182114640 1 --2- 172.21.15.64:0/4184124758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c1078d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.248+0000 7fe182114640 1 -- 172.21.15.64:0/4184124758 >> 172.21.15.64:0/4184124758 conn(0x7fe17c102c60 msgr2=0x7fe17c1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.248+0000 7fe182114640 1 -- 172.21.15.64:0/4184124758 shutdown_connections 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.248+0000 7fe182114640 1 -- 172.21.15.64:0/4184124758 wait complete. 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe182114640 1 Processor -- start 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe182114640 1 -- start start 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe182114640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c07db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:51.848 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe182114640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c10ba30 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe17b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c07db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe17b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c07db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42516/0 (socket says 172.21.15.64:42516) 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe17b7fe640 1 -- 172.21.15.64:0/1681737994 learned_addr learned my addr 172.21.15.64:0/1681737994 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.249+0000 7fe17b7fe640 1 -- 172.21.15.64:0/1681737994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c07e070 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe17b7fe640 1 --2- 172.21.15.64:0/1681737994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c07db30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0097f0 tx=0x7fe16c030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe178ff9640 1 -- 172.21.15.64:0/1681737994 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe16c042070 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe178ff9640 1 -- 172.21.15.64:0/1681737994 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe16c032ca0 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c07c260 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe178ff9640 1 -- 172.21.15.64:0/1681737994 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe16c037040 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c07c7a0 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe178ff9640 1 -- 172.21.15.64:0/1681737994 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe16c045070 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.250+0000 7fe178ff9640 1 -- 172.21.15.64:0/1681737994 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe16c047a40 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.251+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe17c107950 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.253+0000 7fe178ff9640 1 -- 172.21.15.64:0/1681737994 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7fe16c03da60 con 0x7fe17c1074d0 2024-08-19T01:23:51.849 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.312+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe17c07ccd0 con 0x7fe17c1074d0 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.312+0000 7fe178ff9640 1 -- 172.21.15.64:0/1681737994 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fe16c03d430 con 0x7fe17c1074d0 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.313+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 msgr2=0x7fe17c07db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.313+0000 7fe182114640 1 --2- 172.21.15.64:0/1681737994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c07db30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0097f0 tx=0x7fe16c030ec0 comp rx=0 tx=0).stop 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.313+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 shutdown_connections 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.313+0000 7fe182114640 1 --2- 172.21.15.64:0/1681737994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c1074d0 0x7fe17c07db30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.313+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 >> 172.21.15.64:0/1681737994 conn(0x7fe17c102c60 msgr2=0x7fe17c1037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.313+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 shutdown_connections 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:51.313+0000 7fe182114640 1 -- 172.21.15.64:0/1681737994 wait complete. 2024-08-19T01:23:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:Restarting the monitor... 2024-08-19T01:23:51.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:51 smithi064 ceph-mon[18148]: from='client.? 172.21.15.64:0/1681737994' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:23:51.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:51 smithi064 systemd[1]: Stopping Ceph mon.a for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:23:52.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a[18144]: 2024-08-19T01:23:52.132+0000 7fc1d5b43640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:23:52.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a[18144]: 2024-08-19T01:23:52.132+0000 7fc1d5b43640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-19T01:23:52.471 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 podman[18391]: 2024-08-19 01:23:52.175164408 +0000 UTC m=+0.147241894 container died 42aa66ec7591c638e7535caa510d879f169e801c8db68ea9fd2691e523ee354d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-19T01:23:52.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 podman[18391]: 2024-08-19 01:23:52.567183833 +0000 UTC m=+0.539261312 container cleanup 42aa66ec7591c638e7535caa510d879f169e801c8db68ea9fd2691e523ee354d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:23:52.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 bash[18391]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a 2024-08-19T01:23:53.144 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 podman[18403]: 2024-08-19 01:23:52.809891926 +0000 UTC m=+0.626570275 container remove 42aa66ec7591c638e7535caa510d879f169e801c8db68ea9fd2691e523ee354d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-19T01:23:53.145 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-19T01:23:53.145 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 systemd[1]: Stopped Ceph mon.a for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:23:53.145 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:52 smithi064 systemd[1]: Starting Ceph mon.a for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:23:53.454 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 podman[18496]: 2024-08-19 01:23:53.045751714 +0000 UTC m=+0.016961832 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:23:53.455 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 podman[18496]: 2024-08-19 01:23:53.244146657 +0000 UTC m=+0.215356774 container create de0cedb2b02e2950d14257a9b359cdde999dce0a18f4703586f571d9f6f2d43d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-08-19T01:23:53.455 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 podman[18496]: 2024-08-19 01:23:53.453565118 +0000 UTC m=+0.424775252 container init de0cedb2b02e2950d14257a9b359cdde999dce0a18f4703586f571d9f6f2d43d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-19T01:23:53.536 INFO:teuthology.orchestra.run.smithi064.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 podman[18496]: 2024-08-19 01:23:53.456772446 +0000 UTC m=+0.427982564 container start de0cedb2b02e2950d14257a9b359cdde999dce0a18f4703586f571d9f6f2d43d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: set uid:gid to 167:167 (ceph:ceph) 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc), process ceph-mon, pid 2 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: pidfile_write: ignore empty --pid-file 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: load: jerasure load: lrc 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: RocksDB version: 7.9.2 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Git sha 0 2024-08-19T01:23:53.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Compile date 2024-08-18 15:19:56 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: DB SUMMARY 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: DB Session ID: LLO96AJ0UWEYMRPQQ7TN 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: CURRENT file: CURRENT 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: IDENTITY file: IDENTITY 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106416 ; 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.error_if_exists: 0 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.create_if_missing: 0 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.paranoid_checks: 1 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-19T01:23:53.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.env: 0x55f7c22087a0 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.fs: PosixFileSystem 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.info_log: 0x55f7c444d500 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_file_opening_threads: 16 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.statistics: (nil) 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.use_fsync: 0 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_log_file_size: 0 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-19T01:23:53.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.keep_log_file_num: 1000 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.recycle_log_file_num: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.allow_fallocate: 1 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.allow_mmap_reads: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.allow_mmap_writes: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.use_direct_reads: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.create_missing_column_families: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.db_log_dir: 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.wal_dir: 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-19T01:23:53.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.advise_random_on_open: 1 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.db_write_buffer_size: 0 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.write_buffer_manager: 0x55f7c445cdc0 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.rate_limiter: (nil) 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.wal_recovery_mode: 2 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.enable_thread_tracking: 0 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.enable_pipelined_write: 0 2024-08-19T01:23:53.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.unordered_write: 0 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.row_cache: None 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.wal_filter: None 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.allow_ingest_behind: 0 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.two_write_queues: 0 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.manual_wal_flush: 0 2024-08-19T01:23:53.741 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.wal_compression: 0 2024-08-19T01:23:53.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.atomic_flush: 0 2024-08-19T01:23:53.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-19T01:23:53.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-19T01:23:53.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-19T01:23:53.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.log_readahead_size: 0 2024-08-19T01:23:53.743 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.best_efforts_recovery: 0 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.allow_data_in_errors: 0 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.db_host_id: __hostname__ 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_background_jobs: 2 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_background_compactions: -1 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_subcompactions: 1 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-19T01:23:53.744 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_total_wal_size: 0 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_open_files: -1 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bytes_per_sync: 0 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_readahead_size: 0 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_background_flushes: -1 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Compression algorithms supported: 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kZSTD supported: 0 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kXpressCompression supported: 0 2024-08-19T01:23:53.745 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kBZip2Compression supported: 0 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kLZ4Compression supported: 1 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kZlibCompression supported: 1 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kLZ4HCCompression supported: 1 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: kSnappyCompression supported: 1 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.merge_operator: 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_filter: None 2024-08-19T01:23:53.746 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_filter_factory: None 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.sst_partitioner_factory: None 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f7c444d120) 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 4 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2024-08-19T01:23:53.747 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x55f7c4441090 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2024-08-19T01:23:53.748 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: bloomfilter 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 5 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: max_auto_readahead_size: 262144 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: prepopulate_block_cache: 0 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: initial_auto_readahead_size: 8192 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.write_buffer_size: 33554432 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_write_buffer_number: 2 2024-08-19T01:23:53.749 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression: NoCompression 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression: Disabled 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.prefix_extractor: nullptr 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.num_levels: 7 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-19T01:23:53.750 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.level: 32767 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.strategy: 0 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.enabled: false 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-19T01:23:53.751 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.target_file_size_base: 67108864 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-19T01:23:53.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.arena_block_size: 1048576 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.disable_auto_compactions: 0 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-19T01:23:53.753 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.inplace_update_support: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.bloom_locality: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.max_successive_merges: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.paranoid_file_checks: 0 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.force_consistency_checks: 1 2024-08-19T01:23:53.754 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.report_bg_io_stats: 0 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.ttl: 2592000 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.enable_blob_files: false 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.min_blob_size: 0 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.blob_file_size: 268435456 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.blob_file_starting_level: 0 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-19T01:23:53.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dd0f950e-e7f4-4999-9c36-a51bf0c2473e 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724030633483088, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724030633484528, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101436, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 99312, "index_size": 316, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14564, "raw_average_key_size": 50, "raw_value_size": 91387, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 291, "num_filter_entries": 291, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1724030633, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dd0f950e-e7f4-4999-9c36-a51bf0c2473e", "db_session_id": "LLO96AJ0UWEYMRPQQ7TN", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724030633484660, "job": 1, "event": "recovery_finished"} 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f7c446ee00 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: DB pointer 0x55f7c4568000 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-19T01:23:53.756 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 100.92 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T01:23:53.757 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 100.92 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-19T01:23:53.758 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 20.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 20.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache BinnedLRUCache@0x55f7c4441090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: starting mon.a rank 0 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???) e1 preinit fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???).mds e1 new map 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???).mds e1 print_map 2024-08-19T01:23:53.759 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: btime 2024-08-19T01:23:48:740524+0000 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 bash[18496]: de0cedb2b02e2950d14257a9b359cdde999dce0a18f4703586f571d9f6f2d43d 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 systemd[1]: Started Ceph mon.a for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: monmap epoch 1 2024-08-19T01:23:53.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: last_changed 2024-08-19T01:23:45.058123+0000 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: created 2024-08-19T01:23:45.058123+0000 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: min_mon_release 19 (squid) 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: election_strategy: 1 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: fsmap 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: osdmap e1: 0 total, 0 up, 0 in 2024-08-19T01:23:53.761 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:53 smithi064 ceph-mon[18515]: mgrmap e1: no daemons active 2024-08-19T01:23:54.544 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.963+0000 7fb799b7b640 1 Processor -- start 2024-08-19T01:23:54.544 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.963+0000 7fb799b7b640 1 -- start start 2024-08-19T01:23:54.544 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.964+0000 7fb799b7b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb794106770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:54.544 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.964+0000 7fb799b7b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb794106d40 con 0x7fb794106370 2024-08-19T01:23:54.544 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.964+0000 7fb7937fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb794106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:54.544 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.964+0000 7fb7937fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb794106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42528/0 (socket says 172.21.15.64:42528) 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.964+0000 7fb7937fe640 1 -- 172.21.15.64:0/253495918 learned_addr learned my addr 172.21.15.64:0/253495918 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.964+0000 7fb7937fe640 1 -- 172.21.15.64:0/253495918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb794107570 con 0x7fb794106370 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.966+0000 7fb7937fe640 1 --2- 172.21.15.64:0/253495918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb794106770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb784009e80 tx=0x7fb78402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74b1d06cb9476c3 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.966+0000 7fb7927fc640 1 -- 172.21.15.64:0/253495918 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb784036070 con 0x7fb794106370 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.966+0000 7fb7927fc640 1 -- 172.21.15.64:0/253495918 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb7840318c0 con 0x7fb794106370 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.966+0000 7fb7927fc640 1 -- 172.21.15.64:0/253495918 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb784007ab0 con 0x7fb794106370 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 -- 172.21.15.64:0/253495918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 msgr2=0x7fb794106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 --2- 172.21.15.64:0/253495918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb794106770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb784009e80 tx=0x7fb78402a7a0 comp rx=0 tx=0).stop 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 -- 172.21.15.64:0/253495918 shutdown_connections 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 --2- 172.21.15.64:0/253495918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb794106770 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 -- 172.21.15.64:0/253495918 >> 172.21.15.64:0/253495918 conn(0x7fb794101ac0 msgr2=0x7fb794103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 -- 172.21.15.64:0/253495918 shutdown_connections 2024-08-19T01:23:54.545 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 -- 172.21.15.64:0/253495918 wait complete. 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.967+0000 7fb799b7b640 1 Processor -- start 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb799b7b640 1 -- start start 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb799b7b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb79419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb799b7b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb79410a920 con 0x7fb794106370 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb7937fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb79419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb7937fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb79419a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42532/0 (socket says 172.21.15.64:42532) 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb7937fe640 1 -- 172.21.15.64:0/2709874806 learned_addr learned my addr 172.21.15.64:0/2709874806 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb7937fe640 1 -- 172.21.15.64:0/2709874806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79419a900 con 0x7fb794106370 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.968+0000 7fb7937fe640 1 --2- 172.21.15.64:0/2709874806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb79419a3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb784007ce0 tx=0x7fb784031ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.969+0000 7fb790ff9640 1 -- 172.21.15.64:0/2709874806 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb784036050 con 0x7fb794106370 2024-08-19T01:23:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.969+0000 7fb790ff9640 1 -- 172.21.15.64:0/2709874806 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb784042070 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.969+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb79419ab90 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.969+0000 7fb790ff9640 1 -- 172.21.15.64:0/2709874806 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb78403d8a0 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.969+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb79419d880 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.969+0000 7fb790ff9640 1 -- 172.21.15.64:0/2709874806 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb78403db50 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.969+0000 7fb790ff9640 1 -- 172.21.15.64:0/2709874806 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb784007e00 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.970+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb79410aaa0 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:53.972+0000 7fb790ff9640 1 -- 172.21.15.64:0/2709874806 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7fb784007310 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.030+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fb7941067f0 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.035+0000 7fb790ff9640 1 -- 172.21.15.64:0/2709874806 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fb7941067f0 con 0x7fb794106370 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.036+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 msgr2=0x7fb79419a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.036+0000 7fb799b7b640 1 --2- 172.21.15.64:0/2709874806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb79419a3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb784007ce0 tx=0x7fb784031ca0 comp rx=0 tx=0).stop 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.036+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 shutdown_connections 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.036+0000 7fb799b7b640 1 --2- 172.21.15.64:0/2709874806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb794106370 0x7fb79419a3c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:54.547 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.036+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 >> 172.21.15.64:0/2709874806 conn(0x7fb794101ac0 msgr2=0x7fb7941026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:54.548 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.036+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 shutdown_connections 2024-08-19T01:23:54.548 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:54.036+0000 7fb799b7b640 1 -- 172.21.15.64:0/2709874806 wait complete. 2024-08-19T01:23:54.548 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote config to /etc/ceph/ceph.conf 2024-08-19T01:23:54.562 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:23:54.562 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mgr... 2024-08-19T01:23:54.562 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:9283 ... 2024-08-19T01:23:54.563 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:8765 ... 2024-08-19T01:23:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.x 2024-08-19T01:23:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.x.service: Unit ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.x.service not loaded. 2024-08-19T01:23:54.961 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968.target.wants/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@.service. 2024-08-19T01:23:55.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:55 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2709874806' entity='client.admin' 2024-08-19T01:23:55.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 podman[18727]: 2024-08-19 01:23:55.099532884 +0000 UTC m=+0.016849840 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T01:23:55.456 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T01:23:55.456 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-08-19T01:23:55.463 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T01:23:55.463 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-08-19T01:23:55.463 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-08-19T01:23:55.463 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-08-19T01:23:55.472 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T01:23:55.472 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-08-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-08-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-08-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-08-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr to start... 2024-08-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr... 2024-08-19T01:23:55.582 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 podman[18727]: 2024-08-19 01:23:55.357342286 +0000 UTC m=+0.274659230 container init ce16acf65597807653cd6526ec776381e234811f503f7420782d6ecda484279b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0) 2024-08-19T01:23:55.582 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 podman[18727]: 2024-08-19 01:23:55.360095019 +0000 UTC m=+0.277411964 container start ce16acf65597807653cd6526ec776381e234811f503f7420782d6ecda484279b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True) 2024-08-19T01:23:55.582 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 bash[18727]: ce16acf65597807653cd6526ec776381e234811f503f7420782d6ecda484279b 2024-08-19T01:23:55.582 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 systemd[1]: Started Ceph mgr.x for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:23:55.891 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:55.578+0000 7f6bb02a0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:23:55.891 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:55.647+0000 7f6bb02a0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:23:55.891 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:55.799+0000 7f6bb02a0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:23:56.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/186546648' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-19T01:23:56.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:55.971+0000 7f6bb02a0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:23:56.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:56.124+0000 7f6bb02a0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:23:56.556 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:56.257+0000 7f6bb02a0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:23:56.556 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:56.389+0000 7f6bb02a0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:23:56.557 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:56.447+0000 7f6bb02a0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-19T01:23:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:56.562 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-19T01:23:56.563 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-19T01:23:56.563 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-19T01:23:56.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "btime": "2024-08-19T01:23:48:740524+0000", 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-19T01:23:56.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-08-19T01:23:48.741322+0000", 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-19T01:23:56.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.887+0000 7f78c5352640 1 Processor -- start 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.887+0000 7f78c5352640 1 -- start start 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.888+0000 7f78c5352640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c010a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.888+0000 7f78c5352640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f78c007bdb0 con 0x7f78c0107f50 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.888+0000 7f78beffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c010a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.888+0000 7f78beffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c010a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42552/0 (socket says 172.21.15.64:42552) 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.888+0000 7f78beffd640 1 -- 172.21.15.64:0/754011097 learned_addr learned my addr 172.21.15.64:0/754011097 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:56.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.888+0000 7f78beffd640 1 -- 172.21.15.64:0/754011097 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c010a880 con 0x7f78c0107f50 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.889+0000 7f78beffd640 1 --2- 172.21.15.64:0/754011097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c010a340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009820 tx=0x7f78b002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72b17229f4eb442 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.889+0000 7f78bdffb640 1 -- 172.21.15.64:0/754011097 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78b0037070 con 0x7f78c0107f50 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.889+0000 7f78bdffb640 1 -- 172.21.15.64:0/754011097 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f78b0032900 con 0x7f78c0107f50 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.889+0000 7f78bdffb640 1 -- 172.21.15.64:0/754011097 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78b0030850 con 0x7f78c0107f50 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.890+0000 7f78c5352640 1 -- 172.21.15.64:0/754011097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 msgr2=0x7f78c010a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.890+0000 7f78c5352640 1 --2- 172.21.15.64:0/754011097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c010a340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009820 tx=0x7f78b002a270 comp rx=0 tx=0).stop 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.890+0000 7f78c5352640 1 -- 172.21.15.64:0/754011097 shutdown_connections 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.890+0000 7f78c5352640 1 --2- 172.21.15.64:0/754011097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c010a340 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.890+0000 7f78c5352640 1 -- 172.21.15.64:0/754011097 >> 172.21.15.64:0/754011097 conn(0x7f78c0101b10 msgr2=0x7f78c0103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.890+0000 7f78c5352640 1 -- 172.21.15.64:0/754011097 shutdown_connections 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.890+0000 7f78c5352640 1 -- 172.21.15.64:0/754011097 wait complete. 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.891+0000 7f78c5352640 1 Processor -- start 2024-08-19T01:23:56.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.891+0000 7f78c5352640 1 -- start start 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.891+0000 7f78c5352640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.891+0000 7f78c5352640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f78c010b860 con 0x7f78c0107f50 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.891+0000 7f78beffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.891+0000 7f78beffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c019d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42562/0 (socket says 172.21.15.64:42562) 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.891+0000 7f78beffd640 1 -- 172.21.15.64:0/186546648 learned_addr learned my addr 172.21.15.64:0/186546648 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.892+0000 7f78beffd640 1 -- 172.21.15.64:0/186546648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c019db60 con 0x7f78c0107f50 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.892+0000 7f78beffd640 1 --2- 172.21.15.64:0/186546648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c019d620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f78b0032ce0 tx=0x7f78b0032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.892+0000 7f78a7fff640 1 -- 172.21.15.64:0/186546648 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78b0042070 con 0x7f78c0107f50 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.892+0000 7f78a7fff640 1 -- 172.21.15.64:0/186546648 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f78b00308c0 con 0x7f78c0107f50 2024-08-19T01:23:56.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.892+0000 7f78a7fff640 1 -- 172.21.15.64:0/186546648 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78b0037040 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.892+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c019ddf0 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.892+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c01a0ae0 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.893+0000 7f78a7fff640 1 -- 172.21.15.64:0/186546648 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f78b0045070 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.893+0000 7f78a7fff640 1 -- 172.21.15.64:0/186546648 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f78b0047c50 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.893+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c01083b0 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.897+0000 7f78a7fff640 1 -- 172.21.15.64:0/186546648 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f78b003c720 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.959+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f78c01a10d0 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.959+0000 7f78a7fff640 1 -- 172.21.15.64:0/186546648 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f78b003c900 con 0x7f78c0107f50 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.960+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 msgr2=0x7f78c019d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:56.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.960+0000 7f78c5352640 1 --2- 172.21.15.64:0/186546648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c019d620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f78b0032ce0 tx=0x7f78b0032d10 comp rx=0 tx=0).stop 2024-08-19T01:23:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.960+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 shutdown_connections 2024-08-19T01:23:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.960+0000 7f78c5352640 1 --2- 172.21.15.64:0/186546648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78c0107f50 0x7f78c019d620 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.960+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 >> 172.21.15.64:0/186546648 conn(0x7f78c0101b10 msgr2=0x7f78c0102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.960+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 shutdown_connections 2024-08-19T01:23:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:55.960+0000 7f78c5352640 1 -- 172.21.15.64:0/186546648 wait complete. 2024-08-19T01:23:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (1/15)... 2024-08-19T01:23:56.828 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:56.763+0000 7f6bb02a0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:23:56.828 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:56.826+0000 7f6bb02a0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:23:57.112 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:56.896+0000 7f6bb02a0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:23:57.113 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.049+0000 7f6bb02a0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:23:57.113 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.110+0000 7f6bb02a0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:23:57.373 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.179+0000 7f6bb02a0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:23:57.374 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.371+0000 7f6bb02a0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:23:57.735 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.434+0000 7f6bb02a0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:23:57.735 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.495+0000 7f6bb02a0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:23:57.735 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.565+0000 7f6bb02a0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:23:58.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.784+0000 7f6bb02a0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:23:58.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:57.845+0000 7f6bb02a0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:23:58.688 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:58 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:58.393+0000 7f6bb02a0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:23:58.985 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:58 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:58.725+0000 7f6bb02a0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:23:59.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:23:59 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3057779789' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-19T01:23:59.308 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:59 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:59.166+0000 7f6bb02a0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:23:59.308 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:59 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:59.225+0000 7f6bb02a0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:23:59.308 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:23:59 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:23:59.286+0000 7f6bb02a0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:23:59.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:23:59.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-08-19T01:23:59.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:23:59.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-08-19T01:23:59.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-19T01:23:59.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-19T01:23:59.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-19T01:23:59.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-19T01:23:59.638 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "btime": "2024-08-19T01:23:48:740524+0000", 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-08-19T01:23:59.639 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-08-19T01:23:48.741322+0000", 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.001+0000 7f1b4f546640 1 Processor -- start 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.002+0000 7f1b4f546640 1 -- start start 2024-08-19T01:23:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.002+0000 7f1b4f546640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4810a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.002+0000 7f1b4f546640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4807bdf0 con 0x7f1b48107f10 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.002+0000 7f1b4d2e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4810a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.002+0000 7f1b4d2e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4810a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57120/0 (socket says 172.21.15.64:57120) 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.002+0000 7f1b4d2e2640 1 -- 172.21.15.64:0/3678411070 learned_addr learned my addr 172.21.15.64:0/3678411070 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.002+0000 7f1b4d2e2640 1 -- 172.21.15.64:0/3678411070 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4810a840 con 0x7f1b48107f10 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.003+0000 7f1b4d2e2640 1 --2- 172.21.15.64:0/3678411070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4810a300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009820 tx=0x7f1b4402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fef5caa579dc3d9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.003+0000 7f1b3ffff640 1 -- 172.21.15.64:0/3678411070 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b44037070 con 0x7f1b48107f10 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.003+0000 7f1b3ffff640 1 -- 172.21.15.64:0/3678411070 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b44032900 con 0x7f1b48107f10 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.003+0000 7f1b3ffff640 1 -- 172.21.15.64:0/3678411070 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b44030850 con 0x7f1b48107f10 2024-08-19T01:23:59.641 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.004+0000 7f1b4f546640 1 -- 172.21.15.64:0/3678411070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 msgr2=0x7f1b4810a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.004+0000 7f1b4f546640 1 --2- 172.21.15.64:0/3678411070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4810a300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009820 tx=0x7f1b4402a270 comp rx=0 tx=0).stop 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.004+0000 7f1b4f546640 1 -- 172.21.15.64:0/3678411070 shutdown_connections 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.004+0000 7f1b4f546640 1 --2- 172.21.15.64:0/3678411070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4810a300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.004+0000 7f1b4f546640 1 -- 172.21.15.64:0/3678411070 >> 172.21.15.64:0/3678411070 conn(0x7f1b48101b10 msgr2=0x7f1b48103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.004+0000 7f1b4f546640 1 -- 172.21.15.64:0/3678411070 shutdown_connections 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.004+0000 7f1b4f546640 1 -- 172.21.15.64:0/3678411070 wait complete. 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4f546640 1 Processor -- start 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4f546640 1 -- start start 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4f546640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4819b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:23:59.642 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4f546640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4810b820 con 0x7f1b48107f10 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4d2e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4819b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4d2e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4819b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57134/0 (socket says 172.21.15.64:57134) 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4d2e2640 1 -- 172.21.15.64:0/3057779789 learned_addr learned my addr 172.21.15.64:0/3057779789 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.005+0000 7f1b4d2e2640 1 -- 172.21.15.64:0/3057779789 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4819b970 con 0x7f1b48107f10 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b4d2e2640 1 --2- 172.21.15.64:0/3057779789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4819b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b44032ce0 tx=0x7f1b44032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b3e7fc640 1 -- 172.21.15.64:0/3057779789 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b44042070 con 0x7f1b48107f10 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4819bc00 con 0x7f1b48107f10 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b3e7fc640 1 -- 172.21.15.64:0/3057779789 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b440308c0 con 0x7f1b48107f10 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b3e7fc640 1 -- 172.21.15.64:0/3057779789 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b44037040 con 0x7f1b48107f10 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b4819e8f0 con 0x7f1b48107f10 2024-08-19T01:23:59.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b3e7fc640 1 -- 172.21.15.64:0/3057779789 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1b44045070 con 0x7f1b48107f10 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.006+0000 7f1b3e7fc640 1 -- 172.21.15.64:0/3057779789 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b44047bc0 con 0x7f1b48107f10 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.007+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b48108370 con 0x7f1b48107f10 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.009+0000 7f1b3e7fc640 1 -- 172.21.15.64:0/3057779789 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f1b4403c7e0 con 0x7f1b48107f10 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.071+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1b4819ee70 con 0x7f1b48107f10 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.071+0000 7f1b3e7fc640 1 -- 172.21.15.64:0/3057779789 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f1b4403c9c0 con 0x7f1b48107f10 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.072+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 msgr2=0x7f1b4819b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.072+0000 7f1b4f546640 1 --2- 172.21.15.64:0/3057779789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4819b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b44032ce0 tx=0x7f1b44032d10 comp rx=0 tx=0).stop 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.072+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 shutdown_connections 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.072+0000 7f1b4f546640 1 --2- 172.21.15.64:0/3057779789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b48107f10 0x7f1b4819b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.072+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 >> 172.21.15.64:0/3057779789 conn(0x7f1b48101b10 msgr2=0x7f1b481026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.072+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 shutdown_connections 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:23:59.072+0000 7f1b4f546640 1 -- 172.21.15.64:0/3057779789 wait complete. 2024-08-19T01:23:59.644 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (2/15)... 2024-08-19T01:24:00.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: Activating manager daemon x 2024-08-19T01:24:00.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: mgrmap e2: x(active, starting, since 0.00293021s) 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: Manager daemon x is now available 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' 2024-08-19T01:24:00.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:00 smithi064 ceph-mon[18515]: from='mgr.14100 172.21.15.64:0/1107216648' entity='mgr.x' 2024-08-19T01:24:02.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:02 smithi064 ceph-mon[18515]: mgrmap e3: x(active, since 1.00662s) 2024-08-19T01:24:02.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:02 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3737388513' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-19T01:24:02.925 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:24:02.925 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-08-19T01:24:02.925 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:24:02.926 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-19T01:24:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-19T01:24:02.928 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-19T01:24:02.937 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-19T01:24:02.937 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-19T01:24:02.937 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-19T01:24:02.937 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:24:02.937 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-19T01:24:02.937 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-19T01:24:02.937 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "btime": "2024-08-19T01:23:48:740524+0000", 2024-08-19T01:24:02.938 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-19T01:24:02.939 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-08-19T01:23:48.741322+0000", 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-19T01:24:02.940 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff75d72a640 1 Processor -- start 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff75d72a640 1 -- start start 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff75d72a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff75d72a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff75807c820 con 0x7ff75807dde0 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff756ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff756ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57212/0 (socket says 172.21.15.64:57212) 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff756ffd640 1 -- 172.21.15.64:0/4229601030 learned_addr learned my addr 172.21.15.64:0/4229601030 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.033+0000 7ff756ffd640 1 -- 172.21.15.64:0/4229601030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75807c9a0 con 0x7ff75807dde0 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.034+0000 7ff756ffd640 1 --2- 172.21.15.64:0/4229601030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75807c2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff748009820 tx=0x7ff74802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64226dca9056d17 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:02.941 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.034+0000 7ff755ffb640 1 -- 172.21.15.64:0/4229601030 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff748037070 con 0x7ff75807dde0 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.034+0000 7ff755ffb640 1 -- 172.21.15.64:0/4229601030 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff748032900 con 0x7ff75807dde0 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.034+0000 7ff755ffb640 1 -- 172.21.15.64:0/4229601030 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff748030850 con 0x7ff75807dde0 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.035+0000 7ff75d72a640 1 -- 172.21.15.64:0/4229601030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 msgr2=0x7ff75807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.035+0000 7ff75d72a640 1 --2- 172.21.15.64:0/4229601030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75807c2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff748009820 tx=0x7ff74802a270 comp rx=0 tx=0).stop 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.035+0000 7ff75d72a640 1 -- 172.21.15.64:0/4229601030 shutdown_connections 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.035+0000 7ff75d72a640 1 --2- 172.21.15.64:0/4229601030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75807c2e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.035+0000 7ff75d72a640 1 -- 172.21.15.64:0/4229601030 >> 172.21.15.64:0/4229601030 conn(0x7ff758102c20 msgr2=0x7ff7581050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:02.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.035+0000 7ff75d72a640 1 -- 172.21.15.64:0/4229601030 shutdown_connections 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.035+0000 7ff75d72a640 1 -- 172.21.15.64:0/4229601030 wait complete. 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff75d72a640 1 Processor -- start 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff75d72a640 1 -- start start 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff75d72a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75819d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff75d72a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff75810dac0 con 0x7ff75807dde0 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff756ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75819d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff756ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75819d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57214/0 (socket says 172.21.15.64:57214) 2024-08-19T01:24:02.943 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff756ffd640 1 -- 172.21.15.64:0/3737388513 learned_addr learned my addr 172.21.15.64:0/3737388513 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.036+0000 7ff756ffd640 1 -- 172.21.15.64:0/3737388513 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75819dc20 con 0x7ff75807dde0 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.037+0000 7ff756ffd640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75819d6e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff748032ce0 tx=0x7ff748032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.037+0000 7ff73ffff640 1 -- 172.21.15.64:0/3737388513 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff748042070 con 0x7ff75807dde0 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.037+0000 7ff73ffff640 1 -- 172.21.15.64:0/3737388513 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff7480308c0 con 0x7ff75807dde0 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.037+0000 7ff73ffff640 1 -- 172.21.15.64:0/3737388513 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff748037040 con 0x7ff75807dde0 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.037+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75819deb0 con 0x7ff75807dde0 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.037+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7581a0ba0 con 0x7ff75807dde0 2024-08-19T01:24:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.038+0000 7ff73ffff640 1 -- 172.21.15.64:0/3737388513 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 3) ==== 46170+0+0 (secure 0 0 0) 0x7ff748045070 con 0x7ff75807dde0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.038+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff75807c360 con 0x7ff75807dde0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.038+0000 7ff73ffff640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7ff740038e10 0x7ff74003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.038+0000 7ff73ffff640 1 -- 172.21.15.64:0/3737388513 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff74806bdb0 con 0x7ff75807dde0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.038+0000 7ff7567fc640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7ff740038e10 0x7ff74003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.039+0000 7ff7567fc640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7ff740038e10 0x7ff74003b2d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0098c0 tx=0x7ff74c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.043+0000 7ff73ffff640 1 -- 172.21.15.64:0/3737388513 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff74804b170 con 0x7ff75807dde0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.285+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff7581a11b0 con 0x7ff75807dde0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.286+0000 7ff73ffff640 1 -- 172.21.15.64:0/3737388513 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7ff74802fd90 con 0x7ff75807dde0 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7ff740038e10 msgr2=0x7ff74003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7ff740038e10 0x7ff74003b2d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0098c0 tx=0x7ff74c006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 msgr2=0x7ff75819d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75819d6e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff748032ce0 tx=0x7ff748032d10 comp rx=0 tx=0).stop 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 shutdown_connections 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7ff740038e10 0x7ff74003b2d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 --2- 172.21.15.64:0/3737388513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff75807dde0 0x7ff75819d6e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 >> 172.21.15.64:0/3737388513 conn(0x7ff758102c20 msgr2=0x7ff758103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 shutdown_connections 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:02.288+0000 7ff75d72a640 1 -- 172.21.15.64:0/3737388513 wait complete. 2024-08-19T01:24:02.946 INFO:teuthology.orchestra.run.smithi064.stdout:mgr is available 2024-08-19T01:24:03.468 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:03 smithi064 ceph-mon[18515]: mgrmap e4: x(active, since 2s) 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-19T01:24:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.281+0000 7fe398fc7640 1 Processor -- start 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.282+0000 7fe398fc7640 1 -- start start 2024-08-19T01:24:04.036 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.282+0000 7fe398fc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe3941078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.282+0000 7fe398fc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe394107ea0 con 0x7fe3941074d0 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.282+0000 7fe39259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe3941078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.282+0000 7fe39259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe3941078d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57216/0 (socket says 172.21.15.64:57216) 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.282+0000 7fe39259c640 1 -- 172.21.15.64:0/63204627 learned_addr learned my addr 172.21.15.64:0/63204627 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.283+0000 7fe39259c640 1 -- 172.21.15.64:0/63204627 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe394108680 con 0x7fe3941074d0 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.283+0000 7fe39259c640 1 --2- 172.21.15.64:0/63204627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe3941078d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe384009820 tx=0x7fe38402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca36191022a2530 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.283+0000 7fe39159a640 1 -- 172.21.15.64:0/63204627 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe384037070 con 0x7fe3941074d0 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.283+0000 7fe39159a640 1 -- 172.21.15.64:0/63204627 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe384032900 con 0x7fe3941074d0 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.283+0000 7fe39159a640 1 -- 172.21.15.64:0/63204627 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe384030850 con 0x7fe3941074d0 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.284+0000 7fe398fc7640 1 -- 172.21.15.64:0/63204627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 msgr2=0x7fe3941078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.284+0000 7fe398fc7640 1 --2- 172.21.15.64:0/63204627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe3941078d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe384009820 tx=0x7fe38402a270 comp rx=0 tx=0).stop 2024-08-19T01:24:04.037 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.284+0000 7fe398fc7640 1 -- 172.21.15.64:0/63204627 shutdown_connections 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.284+0000 7fe398fc7640 1 --2- 172.21.15.64:0/63204627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe3941078d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.284+0000 7fe398fc7640 1 -- 172.21.15.64:0/63204627 >> 172.21.15.64:0/63204627 conn(0x7fe394102c60 msgr2=0x7fe3941050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.284+0000 7fe398fc7640 1 -- 172.21.15.64:0/63204627 shutdown_connections 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.284+0000 7fe398fc7640 1 -- 172.21.15.64:0/63204627 wait complete. 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe398fc7640 1 Processor -- start 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe398fc7640 1 -- start start 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe398fc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe39419b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe398fc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe39410ba30 con 0x7fe3941074d0 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe39259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe39419b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe39259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe39419b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57218/0 (socket says 172.21.15.64:57218) 2024-08-19T01:24:04.038 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe39259c640 1 -- 172.21.15.64:0/2179524034 learned_addr learned my addr 172.21.15.64:0/2179524034 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe39259c640 1 -- 172.21.15.64:0/2179524034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39419b9f0 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.285+0000 7fe39259c640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe39419b4b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe3840097f0 tx=0x7fe384030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.286+0000 7fe38b7fe640 1 -- 172.21.15.64:0/2179524034 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe384042070 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.286+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39419bc80 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.286+0000 7fe38b7fe640 1 -- 172.21.15.64:0/2179524034 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe384032ca0 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.286+0000 7fe38b7fe640 1 -- 172.21.15.64:0/2179524034 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe384037040 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.286+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39419e970 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.287+0000 7fe38b7fe640 1 -- 172.21.15.64:0/2179524034 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fe384045070 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.287+0000 7fe38b7fe640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7fe380038f20 0x7fe38003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.287+0000 7fe38b7fe640 1 -- 172.21.15.64:0/2179524034 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe38406bff0 con 0x7fe3941074d0 2024-08-19T01:24:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.287+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe39410bbb0 con 0x7fe3941074d0 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.287+0000 7fe391d9b640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7fe380038f20 0x7fe38003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.287+0000 7fe391d9b640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7fe380038f20 0x7fe38003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0098c0 tx=0x7fe37c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.292+0000 7fe38b7fe640 1 -- 172.21.15.64:0/2179524034 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe38403d7f0 con 0x7fe3941074d0 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.466+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe394107950 con 0x7fe3941074d0 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.466+0000 7fe38b7fe640 1 -- 172.21.15.64:0/2179524034 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+429 (secure 0 0 0) 0x7fe38403ccd0 con 0x7fe3941074d0 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.468+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7fe380038f20 msgr2=0x7fe38003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.468+0000 7fe398fc7640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7fe380038f20 0x7fe38003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0098c0 tx=0x7fe37c006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.468+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 msgr2=0x7fe39419b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.468+0000 7fe398fc7640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe39419b4b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe3840097f0 tx=0x7fe384030ec0 comp rx=0 tx=0).stop 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.469+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 shutdown_connections 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.469+0000 7fe398fc7640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7fe380038f20 0x7fe38003b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.469+0000 7fe398fc7640 1 --2- 172.21.15.64:0/2179524034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3941074d0 0x7fe39419b4b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:04.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.469+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 >> 172.21.15.64:0/2179524034 conn(0x7fe394102c60 msgr2=0x7fe3941037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:04.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.469+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 shutdown_connections 2024-08-19T01:24:04.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:03.469+0000 7fe398fc7640 1 -- 172.21.15.64:0/2179524034 wait complete. 2024-08-19T01:24:04.041 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling cephadm module... 2024-08-19T01:24:04.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:04 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2179524034' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-19T01:24:05.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:05 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3912894984' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-08-19T01:24:05.735 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ignoring --setuser ceph since I am not root 2024-08-19T01:24:05.736 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ignoring --setgroup ceph since I am not root 2024-08-19T01:24:05.736 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:05.674+0000 7fa67467d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:24:05.997 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.400+0000 7f76fc9b9640 1 Processor -- start 2024-08-19T01:24:05.997 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.400+0000 7f76fc9b9640 1 -- start start 2024-08-19T01:24:05.997 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.400+0000 7f76fc9b9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f81078e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:05.997 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.400+0000 7f76fc9b9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8107eb0 con 0x7f76f81074e0 2024-08-19T01:24:05.997 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.401+0000 7f76f659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f81078e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:05.997 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.401+0000 7f76f659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f81078e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57228/0 (socket says 172.21.15.64:57228) 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.401+0000 7f76f659c640 1 -- 172.21.15.64:0/220879333 learned_addr learned my addr 172.21.15.64:0/220879333 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.401+0000 7f76f659c640 1 -- 172.21.15.64:0/220879333 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f8108640 con 0x7f76f81074e0 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76f659c640 1 --2- 172.21.15.64:0/220879333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f81078e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009e80 tx=0x7f76e802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f0a201e6783ddf server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76f559a640 1 -- 172.21.15.64:0/220879333 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e8036070 con 0x7f76f81074e0 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76f559a640 1 -- 172.21.15.64:0/220879333 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f76e80318c0 con 0x7f76f81074e0 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76f559a640 1 -- 172.21.15.64:0/220879333 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e8007ab0 con 0x7f76f81074e0 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76fc9b9640 1 -- 172.21.15.64:0/220879333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 msgr2=0x7f76f81078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76fc9b9640 1 --2- 172.21.15.64:0/220879333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f81078e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009e80 tx=0x7f76e802a7a0 comp rx=0 tx=0).stop 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76fc9b9640 1 -- 172.21.15.64:0/220879333 shutdown_connections 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76fc9b9640 1 --2- 172.21.15.64:0/220879333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f81078e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76fc9b9640 1 -- 172.21.15.64:0/220879333 >> 172.21.15.64:0/220879333 conn(0x7f76f8102c50 msgr2=0x7f76f81050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:05.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.402+0000 7f76fc9b9640 1 -- 172.21.15.64:0/220879333 shutdown_connections 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.403+0000 7f76fc9b9640 1 -- 172.21.15.64:0/220879333 wait complete. 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.403+0000 7f76fc9b9640 1 Processor -- start 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.403+0000 7f76fc9b9640 1 -- start start 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.403+0000 7f76fc9b9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f807db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.403+0000 7f76fc9b9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f76f810b9f0 con 0x7f76f81074e0 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76f659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f807db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76f659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f807db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57236/0 (socket says 172.21.15.64:57236) 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76f659c640 1 -- 172.21.15.64:0/3912894984 learned_addr learned my addr 172.21.15.64:0/3912894984 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76f659c640 1 -- 172.21.15.64:0/3912894984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f807e070 con 0x7f76f81074e0 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76f659c640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f807db30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f76e802ad90 tx=0x7f76e8031ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e8036050 con 0x7f76f81074e0 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f76e8042070 con 0x7f76f81074e0 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f807c260 con 0x7f76f81074e0 2024-08-19T01:24:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76e803d7e0 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.404+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f807c7a0 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.405+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f76e803da90 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.405+0000 7f76ef7fe640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f76e4038f70 0x7f76e403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.405+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f810bb70 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.405+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f76e8007e00 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.405+0000 7f76f5d9b640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f76e4038f70 0x7f76e403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.406+0000 7f76f5d9b640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f76e4038f70 0x7f76e403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76e00098c0 tx=0x7f76e0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.410+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f76e8039de0 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:04.633+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f76f807cf90 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.430+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f76e806cca0 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.431+0000 7f76ef7fe640 1 -- 172.21.15.64:0/3912894984 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f76e8047980 con 0x7f76f81074e0 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.433+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f76e4038f70 msgr2=0x7f76e403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:06.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.433+0000 7f76fc9b9640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f76e4038f70 0x7f76e403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76e00098c0 tx=0x7f76e0006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.433+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 msgr2=0x7f76f807db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.433+0000 7f76fc9b9640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f807db30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f76e802ad90 tx=0x7f76e8031ca0 comp rx=0 tx=0).stop 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.434+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 shutdown_connections 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.434+0000 7f76fc9b9640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f76e4038f70 0x7f76e403b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.434+0000 7f76fc9b9640 1 --2- 172.21.15.64:0/3912894984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76f81074e0 0x7f76f807db30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.434+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 >> 172.21.15.64:0/3912894984 conn(0x7f76f8102c50 msgr2=0x7f76f8103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.434+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 shutdown_connections 2024-08-19T01:24:06.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:05.434+0000 7f76fc9b9640 1 -- 172.21.15.64:0/3912894984 wait complete. 2024-08-19T01:24:06.068 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:05.744+0000 7fa67467d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:24:06.069 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:05.894+0000 7fa67467d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:24:06.069 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.066+0000 7fa67467d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:24:06.326 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.218+0000 7fa67467d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:24:06.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3912894984' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-08-19T01:24:06.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-mon[18515]: mgrmap e5: x(active, since 5s) 2024-08-19T01:24:06.611 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.348+0000 7fa67467d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:24:06.611 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.482+0000 7fa67467d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:24:06.611 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.542+0000 7fa67467d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:24:07.154 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.864+0000 7fa67467d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:24:07.154 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.928+0000 7fa67467d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:24:07.154 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:06.999+0000 7fa67467d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:24:07.155 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.152+0000 7fa67467d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:24:07.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-08-19T01:24:07.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-08-19T01:24:07.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-19T01:24:07.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-08-19T01:24:07.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-19T01:24:07.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-08-19T01:24:07.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.402+0000 7f8e93838640 1 Processor -- start 2024-08-19T01:24:07.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.402+0000 7f8e93838640 1 -- start start 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.402+0000 7f8e93838640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c106720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.402+0000 7f8e93838640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c106cf0 con 0x7f8e8c106320 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.402+0000 7f8e915d4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.403+0000 7f8e915d4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57274/0 (socket says 172.21.15.64:57274) 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.403+0000 7f8e915d4640 1 -- 172.21.15.64:0/319834824 learned_addr learned my addr 172.21.15.64:0/319834824 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.403+0000 7f8e915d4640 1 -- 172.21.15.64:0/319834824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c107520 con 0x7f8e8c106320 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.403+0000 7f8e915d4640 1 --2- 172.21.15.64:0/319834824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c106720 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009820 tx=0x7f8e8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d0fcfa55de5b73 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e83fff640 1 -- 172.21.15.64:0/319834824 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e88037070 con 0x7f8e8c106320 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e83fff640 1 -- 172.21.15.64:0/319834824 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e88032900 con 0x7f8e8c106320 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e83fff640 1 -- 172.21.15.64:0/319834824 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e88030850 con 0x7f8e8c106320 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e93838640 1 -- 172.21.15.64:0/319834824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 msgr2=0x7f8e8c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e93838640 1 --2- 172.21.15.64:0/319834824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c106720 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009820 tx=0x7f8e8802a270 comp rx=0 tx=0).stop 2024-08-19T01:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e93838640 1 -- 172.21.15.64:0/319834824 shutdown_connections 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e93838640 1 --2- 172.21.15.64:0/319834824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c106720 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.404+0000 7f8e93838640 1 -- 172.21.15.64:0/319834824 >> 172.21.15.64:0/319834824 conn(0x7f8e8c101b10 msgr2=0x7f8e8c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.405+0000 7f8e93838640 1 -- 172.21.15.64:0/319834824 shutdown_connections 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.405+0000 7f8e93838640 1 -- 172.21.15.64:0/319834824 wait complete. 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.405+0000 7f8e93838640 1 Processor -- start 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.405+0000 7f8e93838640 1 -- start start 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e93838640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c191550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e93838640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c10a8d0 con 0x7f8e8c106320 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e915d4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c191550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e915d4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c191550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57280/0 (socket says 172.21.15.64:57280) 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e915d4640 1 -- 172.21.15.64:0/1511999889 learned_addr learned my addr 172.21.15.64:0/1511999889 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e915d4640 1 -- 172.21.15.64:0/1511999889 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c191a90 con 0x7f8e8c106320 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e915d4640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c191550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009940 tx=0x7f8e88032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e827fc640 1 -- 172.21.15.64:0/1511999889 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e88042070 con 0x7f8e8c106320 2024-08-19T01:24:07.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e827fc640 1 -- 172.21.15.64:0/1511999889 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e880308c0 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8c18fc80 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e827fc640 1 -- 172.21.15.64:0/1511999889 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e88037040 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.406+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8c1901c0 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.407+0000 7f8e827fc640 1 -- 172.21.15.64:0/1511999889 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f8e88045070 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.407+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e8c10aa50 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.408+0000 7f8e827fc640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 0x7f8e7803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.408+0000 7f8e827fc640 1 -- 172.21.15.64:0/1511999889 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e8806bfa0 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.408+0000 7f8e90dd3640 1 -- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 msgr2=0x7f8e7803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3149805608 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.408+0000 7f8e90dd3640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 0x7f8e7803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.412+0000 7f8e827fc640 1 -- 172.21.15.64:0/1511999889 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f8e8803cd90 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.609+0000 7f8e90dd3640 1 -- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 msgr2=0x7f8e7803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3149805608 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.609+0000 7f8e90dd3640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 0x7f8e7803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.639+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8e8c1067a0 con 0x7f8e8c106320 2024-08-19T01:24:07.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.640+0000 7f8e827fc640 1 -- 172.21.15.64:0/1511999889 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f8e8803c820 con 0x7f8e8c106320 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 msgr2=0x7f8e7803b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 0x7f8e7803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 msgr2=0x7f8e8c191550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c191550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009940 tx=0x7f8e88032ce0 comp rx=0 tx=0).stop 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 shutdown_connections 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f8e78038f70 0x7f8e7803b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 --2- 172.21.15.64:0/1511999889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8e8c106320 0x7f8e8c191550 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 >> 172.21.15.64:0/1511999889 conn(0x7f8e8c101b10 msgr2=0x7f8e8c102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 shutdown_connections 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:06.642+0000 7f8e93838640 1 -- 172.21.15.64:0/1511999889 wait complete. 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-08-19T01:24:07.222 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 5... 2024-08-19T01:24:07.406 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.215+0000 7fa67467d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:24:07.406 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.287+0000 7fa67467d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:24:07.690 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1511999889' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-19T01:24:07.690 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.487+0000 7fa67467d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:24:07.690 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.552+0000 7fa67467d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:24:07.691 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.617+0000 7fa67467d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:24:07.691 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.688+0000 7fa67467d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:24:07.973 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.909+0000 7fa67467d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:24:07.973 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:07.970+0000 7fa67467d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:24:08.837 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:08.510+0000 7fa67467d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:24:09.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:08.835+0000 7fa67467d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:24:09.735 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:09.269+0000 7fa67467d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:24:09.735 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:09.327+0000 7fa67467d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:24:09.735 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:09.390+0000 7fa67467d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:24:10.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: Active manager daemon x restarted 2024-08-19T01:24:10.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: Activating manager daemon x 2024-08-19T01:24:10.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: osdmap e2: 0 total, 0 up, 0 in 2024-08-19T01:24:10.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: mgrmap e6: x(active, starting, since 0.00344881s) 2024-08-19T01:24:10.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:24:10.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T01:24:10.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T01:24:10.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: Manager daemon x is now available 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:24:10.987 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:10 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:24:12.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-08-19T01:24:12.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-08-19T01:24:12.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.559+0000 7f603d80e640 1 Processor -- start 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.559+0000 7f603d80e640 1 -- start start 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.559+0000 7f603d80e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603810a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.559+0000 7f603d80e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f603807be70 con 0x7f6038107e90 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.559+0000 7f6036ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603810a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.559+0000 7f6036ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603810a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57294/0 (socket says 172.21.15.64:57294) 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.559+0000 7f6036ffd640 1 -- 172.21.15.64:0/1624730602 learned_addr learned my addr 172.21.15.64:0/1624730602 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.560+0000 7f6036ffd640 1 -- 172.21.15.64:0/1624730602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603810a7c0 con 0x7f6038107e90 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.560+0000 7f6036ffd640 1 --2- 172.21.15.64:0/1624730602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603810a280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6028009820 tx=0x7f602802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facb161eeb0a1559 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.560+0000 7f6035ffb640 1 -- 172.21.15.64:0/1624730602 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6028037070 con 0x7f6038107e90 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f6035ffb640 1 -- 172.21.15.64:0/1624730602 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6028032900 con 0x7f6038107e90 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f6035ffb640 1 -- 172.21.15.64:0/1624730602 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6028030850 con 0x7f6038107e90 2024-08-19T01:24:12.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f603d80e640 1 -- 172.21.15.64:0/1624730602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 msgr2=0x7f603810a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f603d80e640 1 --2- 172.21.15.64:0/1624730602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603810a280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6028009820 tx=0x7f602802a270 comp rx=0 tx=0).stop 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f603d80e640 1 -- 172.21.15.64:0/1624730602 shutdown_connections 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f603d80e640 1 --2- 172.21.15.64:0/1624730602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603810a280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f603d80e640 1 -- 172.21.15.64:0/1624730602 >> 172.21.15.64:0/1624730602 conn(0x7f6038101b10 msgr2=0x7f6038103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f603d80e640 1 -- 172.21.15.64:0/1624730602 shutdown_connections 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.561+0000 7f603d80e640 1 -- 172.21.15.64:0/1624730602 wait complete. 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f603d80e640 1 Processor -- start 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f603d80e640 1 -- start start 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f603d80e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603819b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f603d80e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f603810b860 con 0x7f6038107e90 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f6036ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603819b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f6036ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603819b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57308/0 (socket says 172.21.15.64:57308) 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f6036ffd640 1 -- 172.21.15.64:0/3206471527 learned_addr learned my addr 172.21.15.64:0/3206471527 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:12.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.562+0000 7f6036ffd640 1 -- 172.21.15.64:0/3206471527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603819b980 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.563+0000 7f6036ffd640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603819b440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60280097f0 tx=0x7f6028030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.563+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6028037040 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.563+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6028040070 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.563+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603819bc10 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.563+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f602803a390 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.563+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f603819e900 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.564+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f602803a5e0 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.564+0000 7f603c80c640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 0x7f602003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.564+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 --> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f602003bb90 con 0x7f6020038fc0 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.564+0000 7f60367fc640 1 -- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 msgr2=0x7f602003b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3149805608 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.564+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f602806c120 con 0x7f6038107e90 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.564+0000 7f60367fc640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 0x7f602003b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.765+0000 7f60367fc640 1 -- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 msgr2=0x7f602003b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3149805608 2024-08-19T01:24:12.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:07.765+0000 7f60367fc640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 0x7f602003b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:08.166+0000 7f60367fc640 1 -- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 msgr2=0x7f602003b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3149805608 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:08.166+0000 7f60367fc640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 0x7f602003b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:08.967+0000 7f60367fc640 1 -- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 msgr2=0x7f602003b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3149805608 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:08.967+0000 7f60367fc640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 0x7f602003b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:10.526+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f602806af20 con 0x7f6038107e90 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:10.526+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 msgr2=0x7f602003b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:10.526+0000 7f603c80c640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 0x7f602003b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.527+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f602802fd40 con 0x7f6038107e90 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.528+0000 7f603c80c640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f602003c8f0 0x7f602003ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.528+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6028046ca0 con 0x7f602003c8f0 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.528+0000 7f60367fc640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f602003c8f0 0x7f602003ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.528+0000 7f60367fc640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f602003c8f0 0x7f602003ece0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f602c003980 tx=0x7f602c0092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:12.031 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.529+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f6028046ca0 con 0x7f602003c8f0 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.536+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f60381082f0 con 0x7f602003c8f0 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.536+0000 7f603c80c640 1 -- 172.21.15.64:0/3206471527 <== mgr.14118 v2:172.21.15.64:6800/2403632526 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f60381082f0 con 0x7f602003c8f0 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f602003c8f0 msgr2=0x7f602003ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f602003c8f0 0x7f602003ece0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f602c003980 tx=0x7f602c0092b0 comp rx=0 tx=0).stop 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 msgr2=0x7f603819b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603819b440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60280097f0 tx=0x7f6028030d80 comp rx=0 tx=0).stop 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 shutdown_connections 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f602003c8f0 0x7f602003ece0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:6800/3149805608,v1:172.21.15.64:6801/3149805608] conn(0x7f6020038fc0 0x7f602003b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 --2- 172.21.15.64:0/3206471527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6038107e90 0x7f603819b440 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 >> 172.21.15.64:0/3206471527 conn(0x7f6038101b10 msgr2=0x7f6038102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 shutdown_connections 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:11.537+0000 7f603d80e640 1 -- 172.21.15.64:0/3206471527 wait complete. 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 5 is available 2024-08-19T01:24:12.032 INFO:teuthology.orchestra.run.smithi064.stdout:Setting orchestrator backend to cephadm... 2024-08-19T01:24:12.410 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:12 smithi064 ceph-mon[18515]: Found migration_current of "None". Setting to last migration. 2024-08-19T01:24:12.410 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:12 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:12.410 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:12 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:12.410 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:12 smithi064 ceph-mon[18515]: mgrmap e7: x(active, since 1.00577s) 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.408+0000 7f86aec05640 1 Processor -- start 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.408+0000 7f86aec05640 1 -- start start 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.408+0000 7f86aec05640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.408+0000 7f86aec05640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86a8106d00 con 0x7f86a8106330 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.408+0000 7f86ac9a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.408+0000 7f86ac9a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34892/0 (socket says 172.21.15.64:34892) 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.408+0000 7f86ac9a1640 1 -- 172.21.15.64:0/3996869197 learned_addr learned my addr 172.21.15.64:0/3996869197 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.409+0000 7f86ac9a1640 1 -- 172.21.15.64:0/3996869197 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a8107530 con 0x7f86a8106330 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.409+0000 7f86ac9a1640 1 --2- 172.21.15.64:0/3996869197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8106730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f869c009820 tx=0x7f869c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d775905f70f23cb server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:13.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.409+0000 7f86a77fe640 1 -- 172.21.15.64:0/3996869197 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c037070 con 0x7f86a8106330 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.409+0000 7f86a77fe640 1 -- 172.21.15.64:0/3996869197 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f869c032900 con 0x7f86a8106330 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86a77fe640 1 -- 172.21.15.64:0/3996869197 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c030850 con 0x7f86a8106330 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86aec05640 1 -- 172.21.15.64:0/3996869197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 msgr2=0x7f86a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86aec05640 1 --2- 172.21.15.64:0/3996869197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8106730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f869c009820 tx=0x7f869c02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86aec05640 1 -- 172.21.15.64:0/3996869197 shutdown_connections 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86aec05640 1 --2- 172.21.15.64:0/3996869197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8106730 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86aec05640 1 -- 172.21.15.64:0/3996869197 >> 172.21.15.64:0/3996869197 conn(0x7f86a8101ac0 msgr2=0x7f86a8103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86aec05640 1 -- 172.21.15.64:0/3996869197 shutdown_connections 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.410+0000 7f86aec05640 1 -- 172.21.15.64:0/3996869197 wait complete. 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86aec05640 1 Processor -- start 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86aec05640 1 -- start start 2024-08-19T01:24:13.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86aec05640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8191560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86aec05640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86a810a8e0 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86ac9a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8191560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86ac9a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8191560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34896/0 (socket says 172.21.15.64:34896) 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86ac9a1640 1 -- 172.21.15.64:0/3270715392 learned_addr learned my addr 172.21.15.64:0/3270715392 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86ac9a1640 1 -- 172.21.15.64:0/3270715392 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a8191aa0 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.411+0000 7f86ac9a1640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8191560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f869c0097f0 tx=0x7f869c030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.412+0000 7f86a5ffb640 1 -- 172.21.15.64:0/3270715392 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c037040 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.412+0000 7f86a5ffb640 1 -- 172.21.15.64:0/3270715392 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f869c040070 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.412+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a818fc90 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.412+0000 7f86a5ffb640 1 -- 172.21.15.64:0/3270715392 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c03a390 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.412+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a8190170 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.413+0000 7f86a5ffb640 1 -- 172.21.15.64:0/3270715392 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f869c03b430 con 0x7f86a8106330 2024-08-19T01:24:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.413+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a810aa60 con 0x7f86a8106330 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.413+0000 7f86a5ffb640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f8698038eb0 0x7f869803b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.413+0000 7f86a5ffb640 1 -- 172.21.15.64:0/3270715392 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f869c06c260 con 0x7f86a8106330 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.413+0000 7f86a7fff640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f8698038eb0 0x7f869803b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.414+0000 7f86a7fff640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f8698038eb0 0x7f869803b370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f86940098c0 tx=0x7f8694006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.418+0000 7f86a5ffb640 1 -- 172.21.15.64:0/3270715392 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f869c032900 con 0x7f86a8106330 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.612+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f86a8195330 con 0x7f8698038eb0 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.617+0000 7f86a5ffb640 1 -- 172.21.15.64:0/3270715392 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f86a8195330 con 0x7f8698038eb0 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f8698038eb0 msgr2=0x7f869803b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f8698038eb0 0x7f869803b370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f86940098c0 tx=0x7f8694006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 msgr2=0x7f86a8191560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:13.176 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8191560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f869c0097f0 tx=0x7f869c030d80 comp rx=0 tx=0).stop 2024-08-19T01:24:13.177 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 shutdown_connections 2024-08-19T01:24:13.177 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f8698038eb0 0x7f869803b370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:13.177 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 --2- 172.21.15.64:0/3270715392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86a8106330 0x7f86a8191560 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:13.177 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 >> 172.21.15.64:0/3270715392 conn(0x7f86a8101ac0 msgr2=0x7f86a8102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:13.177 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 shutdown_connections 2024-08-19T01:24:13.177 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:12.620+0000 7f86aec05640 1 -- 172.21.15.64:0/3270715392 wait complete. 2024-08-19T01:24:13.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-mon[18515]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-19T01:24:13.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-mon[18515]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:13.268+0000 7fa5dbb4b640 -1 log_channel(cephadm) log [ERR] : [19/Aug/2024:01:24:13] ENGINE Error in HTTPServer.serve 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: Traceback (most recent call last): 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._connections.run(self.expiration_interval) 2024-08-19T01:24:13.565 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._run(expiration_interval) 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: s = self.context.wrap_socket( 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: return self.sslsocket_class._create( 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self.do_handshake() 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:24:13.566 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._sslobj.do_handshake() 2024-08-19T01:24:13.567 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:24:13.567 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout value unchanged 2024-08-19T01:24:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.562+0000 7f4493330640 1 Processor -- start 2024-08-19T01:24:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.562+0000 7f4493330640 1 -- start start 2024-08-19T01:24:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.562+0000 7f4493330640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c106720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.562+0000 7f4493330640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f448c106cf0 con 0x7f448c106320 2024-08-19T01:24:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.563+0000 7f44910cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.563+0000 7f44910cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34904/0 (socket says 172.21.15.64:34904) 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.563+0000 7f44910cc640 1 -- 172.21.15.64:0/1052695509 learned_addr learned my addr 172.21.15.64:0/1052695509 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.563+0000 7f44910cc640 1 -- 172.21.15.64:0/1052695509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448c107520 con 0x7f448c106320 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f44910cc640 1 --2- 172.21.15.64:0/1052695509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c106720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f448800bac0 tx=0x7f448802c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96e73ce7f33e6c5 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4483fff640 1 -- 172.21.15.64:0/1052695509 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44880326d0 con 0x7f448c106320 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4483fff640 1 -- 172.21.15.64:0/1052695509 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4488038070 con 0x7f448c106320 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4483fff640 1 -- 172.21.15.64:0/1052695509 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4488033d70 con 0x7f448c106320 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4493330640 1 -- 172.21.15.64:0/1052695509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 msgr2=0x7f448c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4493330640 1 --2- 172.21.15.64:0/1052695509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c106720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f448800bac0 tx=0x7f448802c920 comp rx=0 tx=0).stop 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4493330640 1 -- 172.21.15.64:0/1052695509 shutdown_connections 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4493330640 1 --2- 172.21.15.64:0/1052695509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c106720 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.564+0000 7f4493330640 1 -- 172.21.15.64:0/1052695509 >> 172.21.15.64:0/1052695509 conn(0x7f448c101b10 msgr2=0x7f448c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f4493330640 1 -- 172.21.15.64:0/1052695509 shutdown_connections 2024-08-19T01:24:14.339 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f4493330640 1 -- 172.21.15.64:0/1052695509 wait complete. 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f4493330640 1 Processor -- start 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f4493330640 1 -- start start 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f4493330640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f4493330640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f448c07b110 con 0x7f448c106320 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f44910cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.565+0000 7f44910cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34918/0 (socket says 172.21.15.64:34918) 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f44910cc640 1 -- 172.21.15.64:0/4117004613 learned_addr learned my addr 172.21.15.64:0/4117004613 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f44910cc640 1 -- 172.21.15.64:0/4117004613 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448c199760 con 0x7f448c106320 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f44910cc640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c199220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f448800b2c0 tx=0x7f44880327d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f44827fc640 1 -- 172.21.15.64:0/4117004613 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4488042070 con 0x7f448c106320 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f448c1999f0 con 0x7f448c106320 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f44827fc640 1 -- 172.21.15.64:0/4117004613 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4488032b70 con 0x7f448c106320 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f44827fc640 1 -- 172.21.15.64:0/4117004613 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f448803c580 con 0x7f448c106320 2024-08-19T01:24:14.340 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.566+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f448c19c6e0 con 0x7f448c106320 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.567+0000 7f44827fc640 1 -- 172.21.15.64:0/4117004613 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f448803c840 con 0x7f448c106320 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.567+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f448c106720 con 0x7f448c106320 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.567+0000 7f44827fc640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f4478038f70 0x7f447803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.567+0000 7f44827fc640 1 -- 172.21.15.64:0/4117004613 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4488030050 con 0x7f448c106320 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.567+0000 7f44908cb640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f4478038f70 0x7f447803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.568+0000 7f44908cb640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f4478038f70 0x7f447803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f447c0098c0 tx=0x7f447c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.572+0000 7f44827fc640 1 -- 172.21.15.64:0/4117004613 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f4488032d90 con 0x7f448c106320 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.751+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f448c062360 con 0x7f4478038f70 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.751+0000 7f44827fc640 1 -- 172.21.15.64:0/4117004613 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f448c062360 con 0x7f4478038f70 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f4478038f70 msgr2=0x7f447803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f4478038f70 0x7f447803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f447c0098c0 tx=0x7f447c006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 msgr2=0x7f448c199220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:14.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c199220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f448800b2c0 tx=0x7f44880327d0 comp rx=0 tx=0).stop 2024-08-19T01:24:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 shutdown_connections 2024-08-19T01:24:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f4478038f70 0x7f447803b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 --2- 172.21.15.64:0/4117004613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f448c106320 0x7f448c199220 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 >> 172.21.15.64:0/4117004613 conn(0x7f448c101b10 msgr2=0x7f448c18f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 shutdown_connections 2024-08-19T01:24:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:13.753+0000 7f4493330640 1 -- 172.21.15.64:0/4117004613 wait complete. 2024-08-19T01:24:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:Generating ssh key... 2024-08-19T01:24:14.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:14 smithi064 ceph-mon[18515]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:14.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:14 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:13] ENGINE Bus STARTING 2024-08-19T01:24:14.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:14 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:13] ENGINE Serving on http://172.21.15.64:8765 2024-08-19T01:24:14.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:14 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:13] ENGINE Serving on https://172.21.15.64:7150 2024-08-19T01:24:14.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:14 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:13] ENGINE Bus STARTED 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:14 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:13] ENGINE Error in HTTPServer.serve 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._connections.run(self.expiration_interval) 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._run(expiration_interval) 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: s = self.context.wrap_socket( 2024-08-19T01:24:14.589 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:24:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout: return self.sslsocket_class._create( 2024-08-19T01:24:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:24:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout: self.do_handshake() 2024-08-19T01:24:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:24:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._sslobj.do_handshake() 2024-08-19T01:24:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:24:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:14 smithi064 ceph-mon[18515]: mgrmap e8: x(active, since 2s) 2024-08-19T01:24:15.236 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: Generating public/private rsa key pair. 2024-08-19T01:24:15.236 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: Your identification has been saved in /tmp/tmpnjwrbkg8/key 2024-08-19T01:24:15.236 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: Your public key has been saved in /tmp/tmpnjwrbkg8/key.pub 2024-08-19T01:24:15.236 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: The key fingerprint is: 2024-08-19T01:24:15.236 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: SHA256:6/7cQnJSWhcLhAeCkzjFZt0IH07DJ9EDtDvyz2ZfTgE ceph-88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:24:15.237 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: The key's randomart image is: 2024-08-19T01:24:15.237 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: +---[RSA 3072]----+ 2024-08-19T01:24:15.237 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | +oBO*.+. | 2024-08-19T01:24:15.237 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | o B+==* o . | 2024-08-19T01:24:15.237 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | + .+o oE. o | 2024-08-19T01:24:15.237 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | . o.o | 2024-08-19T01:24:15.238 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | . o S+ .. | 2024-08-19T01:24:15.238 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | o .+.o . | 2024-08-19T01:24:15.238 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | . .= o | 2024-08-19T01:24:15.238 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | +o..= | 2024-08-19T01:24:15.238 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: | +=o+.o | 2024-08-19T01:24:15.238 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: +----[SHA256]-----+ 2024-08-19T01:24:15.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-mon[18515]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:15.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:15.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:15 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.780+0000 7f36bb8fc640 1 Processor -- start 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.780+0000 7f36bb8fc640 1 -- start start 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.781+0000 7f36bb8fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b410a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.781+0000 7f36bb8fc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f36b407be30 con 0x7f36b4107ed0 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.781+0000 7f36b9698640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b410a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.781+0000 7f36b9698640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b410a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34932/0 (socket says 172.21.15.64:34932) 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.781+0000 7f36b9698640 1 -- 172.21.15.64:0/255657302 learned_addr learned my addr 172.21.15.64:0/255657302 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.781+0000 7f36b9698640 1 -- 172.21.15.64:0/255657302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b410a800 con 0x7f36b4107ed0 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.782+0000 7f36b9698640 1 --2- 172.21.15.64:0/255657302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b410a2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009820 tx=0x7f36ac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe385de3d485bbc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:15.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.782+0000 7f36b3fff640 1 -- 172.21.15.64:0/255657302 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ac037070 con 0x7f36b4107ed0 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.782+0000 7f36b3fff640 1 -- 172.21.15.64:0/255657302 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f36ac032900 con 0x7f36b4107ed0 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.782+0000 7f36b3fff640 1 -- 172.21.15.64:0/255657302 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ac030850 con 0x7f36b4107ed0 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.782+0000 7f36bb8fc640 1 -- 172.21.15.64:0/255657302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 msgr2=0x7f36b410a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.782+0000 7f36bb8fc640 1 --2- 172.21.15.64:0/255657302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b410a2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009820 tx=0x7f36ac02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 -- 172.21.15.64:0/255657302 shutdown_connections 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 --2- 172.21.15.64:0/255657302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b410a2c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 -- 172.21.15.64:0/255657302 >> 172.21.15.64:0/255657302 conn(0x7f36b4101b10 msgr2=0x7f36b4103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 -- 172.21.15.64:0/255657302 shutdown_connections 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 -- 172.21.15.64:0/255657302 wait complete. 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 Processor -- start 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 -- start start 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.783+0000 7f36bb8fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36bb8fc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f36b410b840 con 0x7f36b4107ed0 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b9698640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:15.737 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b9698640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b419b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34938/0 (socket says 172.21.15.64:34938) 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b9698640 1 -- 172.21.15.64:0/1010924504 learned_addr learned my addr 172.21.15.64:0/1010924504 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b9698640 1 -- 172.21.15.64:0/1010924504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b419b9a0 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b9698640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b419b460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f36ac032ce0 tx=0x7f36ac032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b27fc640 1 -- 172.21.15.64:0/1010924504 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ac042070 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b419bc30 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b27fc640 1 -- 172.21.15.64:0/1010924504 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f36ac0308c0 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36b27fc640 1 -- 172.21.15.64:0/1010924504 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ac037040 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.784+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b419e920 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.785+0000 7f36b27fc640 1 -- 172.21.15.64:0/1010924504 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f36ac045070 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.785+0000 7f36b27fc640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f369c038fc0 0x7f369c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.785+0000 7f36b27fc640 1 -- 172.21.15.64:0/1010924504 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f36ac06c050 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.785+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b4108330 con 0x7f36b4107ed0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.785+0000 7f36b8e97640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f369c038fc0 0x7f369c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:15.738 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.786+0000 7f36b8e97640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f369c038fc0 0x7f369c03b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36a80098c0 tx=0x7f36a8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.790+0000 7f36b27fc640 1 -- 172.21.15.64:0/1010924504 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f36ac045360 con 0x7f36b4107ed0 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:14.967+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f36b4102e20 con 0x7f369c038fc0 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.155+0000 7f36b27fc640 1 -- 172.21.15.64:0/1010924504 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f36b4102e20 con 0x7f369c038fc0 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f369c038fc0 msgr2=0x7f369c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f369c038fc0 0x7f369c03b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36a80098c0 tx=0x7f36a8006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 msgr2=0x7f36b419b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b419b460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f36ac032ce0 tx=0x7f36ac032d10 comp rx=0 tx=0).stop 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 shutdown_connections 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f369c038fc0 0x7f369c03b480 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 --2- 172.21.15.64:0/1010924504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b4107ed0 0x7f36b419b460 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 >> 172.21.15.64:0/1010924504 conn(0x7f36b4101b10 msgr2=0x7f36b4102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 shutdown_connections 2024-08-19T01:24:15.739 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:15.157+0000 7f36bb8fc640 1 -- 172.21.15.64:0/1010924504 wait complete. 2024-08-19T01:24:16.651 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:16 smithi064 ceph-mon[18515]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:16.651 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:16 smithi064 ceph-mon[18515]: Generating ssh key... 2024-08-19T01:24:16.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoB69Z0HhChZZnW8q9333aR0h0iqtwWIXtkxv/ZlzkIPTWonq5gxRue5Qiqtl5pLM6KtcnDaFDdZOWG3wS7BRoP679a+23tNRTMycXRbjV9+veH+suPFZLM69zwP2TNE810Yb8LOAXxEqfxxn9+SEa2L5g9hBW9xwkg2yi6V8hrSFzAAx8rmdQfDKzpt3iqB3z4OzI0p+NIjqQAqALlsKabkYHN96KDtnUQ30AauFhISRoyx1o4M8yUdq7kdtABVOnK+4nD9wRXk33aghZ518zbEtOwSBhlExb41skQ8UHGd3Ms4RXoKtbq2olpxml4H/6pCR0WvO5A0Pk338AgYX6wHtN+b/+FuDE5nNsMR7fU3EN+fv9NmobfzFBAUzasLqvFJxf7LVDc68hjIr0H4aJGuYjfjoTaitOnSfo4imT/VvQLk8at6Zlq8QCtGlQD+2anRcXanFwChrRylTyI5zDTlO2wBrKxm/lO/KoBLsijKuFHUHGHZ8fO9L1Z4vg/s= ceph-88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.148+0000 7fb780dc3640 1 Processor -- start 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.148+0000 7fb780dc3640 1 -- start start 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.149+0000 7fb780dc3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c106720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.149+0000 7fb780dc3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c106cf0 con 0x7fb77c106320 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.149+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.149+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34952/0 (socket says 172.21.15.64:34952) 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.149+0000 7fb77a59c640 1 -- 172.21.15.64:0/1583837822 learned_addr learned my addr 172.21.15.64:0/1583837822 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.149+0000 7fb77a59c640 1 -- 172.21.15.64:0/1583837822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c107520 con 0x7fb77c106320 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.150+0000 7fb77a59c640 1 --2- 172.21.15.64:0/1583837822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c106720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009820 tx=0x7fb76c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7faeaec94f7d3a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.150+0000 7fb77959a640 1 -- 172.21.15.64:0/1583837822 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb76c037070 con 0x7fb77c106320 2024-08-19T01:24:16.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.150+0000 7fb77959a640 1 -- 172.21.15.64:0/1583837822 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb76c032900 con 0x7fb77c106320 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.150+0000 7fb77959a640 1 -- 172.21.15.64:0/1583837822 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb76c030850 con 0x7fb77c106320 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 -- 172.21.15.64:0/1583837822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 msgr2=0x7fb77c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 --2- 172.21.15.64:0/1583837822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c106720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009820 tx=0x7fb76c02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 -- 172.21.15.64:0/1583837822 shutdown_connections 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 --2- 172.21.15.64:0/1583837822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c106720 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 -- 172.21.15.64:0/1583837822 >> 172.21.15.64:0/1583837822 conn(0x7fb77c101b10 msgr2=0x7fb77c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 -- 172.21.15.64:0/1583837822 shutdown_connections 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 -- 172.21.15.64:0/1583837822 wait complete. 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.151+0000 7fb780dc3640 1 Processor -- start 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb780dc3640 1 -- start start 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb780dc3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c07ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb780dc3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c10a8d0 con 0x7fb77c106320 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c07ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c07ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34964/0 (socket says 172.21.15.64:34964) 2024-08-19T01:24:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb77a59c640 1 -- 172.21.15.64:0/4218703315 learned_addr learned my addr 172.21.15.64:0/4218703315 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb77a59c640 1 -- 172.21.15.64:0/4218703315 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c19e200 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb77a59c640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c07ba10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009940 tx=0x7fb76c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb7737fe640 1 -- 172.21.15.64:0/4218703315 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb76c042070 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.152+0000 7fb7737fe640 1 -- 172.21.15.64:0/4218703315 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb76c0308c0 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.153+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77c19e3d0 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.153+0000 7fb7737fe640 1 -- 172.21.15.64:0/4218703315 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb76c037040 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.153+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77c19e770 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.153+0000 7fb7737fe640 1 -- 172.21.15.64:0/4218703315 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb76c045070 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.154+0000 7fb7737fe640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fb768038fc0 0x7fb76803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.154+0000 7fb7737fe640 1 -- 172.21.15.64:0/4218703315 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb76c06c030 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.154+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77c1067a0 con 0x7fb77c106320 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.154+0000 7fb779d9b640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fb768038fc0 0x7fb76803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.154+0000 7fb779d9b640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fb768038fc0 0x7fb76803b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb7640098c0 tx=0x7fb764006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.158+0000 7fb7737fe640 1 -- 172.21.15.64:0/4218703315 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb76c045360 con 0x7fb77c106320 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.336+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fb77c102dc0 con 0x7fb768038fc0 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.336+0000 7fb7737fe640 1 -- 172.21.15.64:0/4218703315 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fb77c102dc0 con 0x7fb768038fc0 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fb768038fc0 msgr2=0x7fb76803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fb768038fc0 0x7fb76803b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb7640098c0 tx=0x7fb764006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 msgr2=0x7fb77c07ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c07ba10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009940 tx=0x7fb76c032ce0 comp rx=0 tx=0).stop 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 shutdown_connections 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fb768038fc0 0x7fb76803b480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 --2- 172.21.15.64:0/4218703315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb77c106320 0x7fb77c07ba10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:16.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 >> 172.21.15.64:0/4218703315 conn(0x7fb77c101b10 msgr2=0x7fb77c1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:16.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.338+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 shutdown_connections 2024-08-19T01:24:16.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:16.339+0000 7fb780dc3640 1 -- 172.21.15.64:0/4218703315 wait complete. 2024-08-19T01:24:16.898 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-08-19T01:24:16.898 INFO:teuthology.orchestra.run.smithi064.stdout:Adding key to root@localhost authorized_keys... 2024-08-19T01:24:16.898 INFO:teuthology.orchestra.run.smithi064.stdout:Adding host smithi064... 2024-08-19T01:24:17.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:17 smithi064 ceph-mon[18515]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:18.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:18 smithi064 ceph-mon[18515]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:19.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:19 smithi064 ceph-mon[18515]: Deploying cephadm binary to smithi064 2024-08-19T01:24:21.911 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Added host 'smithi064' with addr '172.21.15.64' 2024-08-19T01:24:21.911 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.266+0000 7f6651103640 1 Processor -- start 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.266+0000 7f6651103640 1 -- start start 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.267+0000 7f6651103640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c106720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.267+0000 7f6651103640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f664c106cf0 con 0x7f664c106320 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.267+0000 7f664ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.267+0000 7f664ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34974/0 (socket says 172.21.15.64:34974) 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.267+0000 7f664ad9d640 1 -- 172.21.15.64:0/3136240991 learned_addr learned my addr 172.21.15.64:0/3136240991 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.267+0000 7f664ad9d640 1 -- 172.21.15.64:0/3136240991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664c107520 con 0x7f664c106320 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.268+0000 7f664ad9d640 1 --2- 172.21.15.64:0/3136240991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c106720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f663c009820 tx=0x7f663c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f314847043b7a57f server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.268+0000 7f664a59c640 1 -- 172.21.15.64:0/3136240991 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f663c037070 con 0x7f664c106320 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.268+0000 7f664a59c640 1 -- 172.21.15.64:0/3136240991 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f663c032900 con 0x7f664c106320 2024-08-19T01:24:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.268+0000 7f664a59c640 1 -- 172.21.15.64:0/3136240991 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f663c030850 con 0x7f664c106320 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.269+0000 7f6651103640 1 -- 172.21.15.64:0/3136240991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 msgr2=0x7f664c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.269+0000 7f6651103640 1 --2- 172.21.15.64:0/3136240991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c106720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f663c009820 tx=0x7f663c02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.269+0000 7f6651103640 1 -- 172.21.15.64:0/3136240991 shutdown_connections 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.269+0000 7f6651103640 1 --2- 172.21.15.64:0/3136240991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c106720 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.269+0000 7f6651103640 1 -- 172.21.15.64:0/3136240991 >> 172.21.15.64:0/3136240991 conn(0x7f664c101b10 msgr2=0x7f664c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.269+0000 7f6651103640 1 -- 172.21.15.64:0/3136240991 shutdown_connections 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.269+0000 7f6651103640 1 -- 172.21.15.64:0/3136240991 wait complete. 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f6651103640 1 Processor -- start 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f6651103640 1 -- start start 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f6651103640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c19b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f6651103640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f664c10a8d0 con 0x7f664c106320 2024-08-19T01:24:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f664ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c19b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f664ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c19b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34978/0 (socket says 172.21.15.64:34978) 2024-08-19T01:24:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f664ad9d640 1 -- 172.21.15.64:0/3046337500 learned_addr learned my addr 172.21.15.64:0/3046337500 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f664ad9d640 1 -- 172.21.15.64:0/3046337500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664c19ba20 con 0x7f664c106320 2024-08-19T01:24:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.270+0000 7f664ad9d640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c19b4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f663c0097f0 tx=0x7f663c030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.271+0000 7f6648d99640 1 -- 172.21.15.64:0/3046337500 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f663c037040 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.271+0000 7f6648d99640 1 -- 172.21.15.64:0/3046337500 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f663c040070 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.271+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f664c19bcb0 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.271+0000 7f6648d99640 1 -- 172.21.15.64:0/3046337500 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f663c03a390 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.271+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f664c19e9a0 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.272+0000 7f6648d99640 1 -- 172.21.15.64:0/3046337500 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f663c03a600 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.272+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f664c10aa50 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.272+0000 7f6648d99640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f6638038fc0 0x7f663803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.272+0000 7f6648d99640 1 -- 172.21.15.64:0/3046337500 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f663c06c450 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.272+0000 7f6643fff640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f6638038fc0 0x7f663803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.272+0000 7f6643fff640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f6638038fc0 0x7f663803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66340098c0 tx=0x7f6634006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.277+0000 7f6648d99640 1 -- 172.21.15.64:0/3046337500 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f663c03a9f0 con 0x7f664c106320 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:17.457+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}) -- 0x7f664c02d860 con 0x7f6638038fc0 2024-08-19T01:24:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.319+0000 7f6648d99640 1 -- 172.21.15.64:0/3046337500 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f664c02d860 con 0x7f6638038fc0 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.321+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f6638038fc0 msgr2=0x7f663803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.321+0000 7f6651103640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f6638038fc0 0x7f663803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66340098c0 tx=0x7f6634006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.321+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 msgr2=0x7f664c19b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.321+0000 7f6651103640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c19b4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f663c0097f0 tx=0x7f663c030d80 comp rx=0 tx=0).stop 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.322+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 shutdown_connections 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.322+0000 7f6651103640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f6638038fc0 0x7f663803b480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.322+0000 7f6651103640 1 --2- 172.21.15.64:0/3046337500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f664c106320 0x7f664c19b4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.322+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 >> 172.21.15.64:0/3046337500 conn(0x7f664c101b10 msgr2=0x7f664c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.322+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 shutdown_connections 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:21.322+0000 7f6651103640 1 -- 172.21.15.64:0/3046337500 wait complete. 2024-08-19T01:24:21.916 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mon service... 2024-08-19T01:24:22.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:22 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:22.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:22 smithi064 ceph-mon[18515]: Added host smithi064 2024-08-19T01:24:22.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:22 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.277+0000 7fd9789a2640 1 Processor -- start 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.277+0000 7fd9789a2640 1 -- start start 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.277+0000 7fd9789a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd9741061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.277+0000 7fd9789a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd97407bda0 con 0x7fd974105df0 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.278+0000 7fd97259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd9741061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.278+0000 7fd97259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd9741061f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40004/0 (socket says 172.21.15.64:40004) 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.278+0000 7fd97259c640 1 -- 172.21.15.64:0/1372087050 learned_addr learned my addr 172.21.15.64:0/1372087050 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.278+0000 7fd97259c640 1 -- 172.21.15.64:0/1372087050 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd974106730 con 0x7fd974105df0 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.279+0000 7fd97259c640 1 --2- 172.21.15.64:0/1372087050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd9741061f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd964009820 tx=0x7fd96402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f521e6db34f2db88 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.279+0000 7fd97159a640 1 -- 172.21.15.64:0/1372087050 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964037070 con 0x7fd974105df0 2024-08-19T01:24:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.279+0000 7fd97159a640 1 -- 172.21.15.64:0/1372087050 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd964032900 con 0x7fd974105df0 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.279+0000 7fd97159a640 1 -- 172.21.15.64:0/1372087050 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964030850 con 0x7fd974105df0 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.280+0000 7fd9789a2640 1 -- 172.21.15.64:0/1372087050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 msgr2=0x7fd9741061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.280+0000 7fd9789a2640 1 --2- 172.21.15.64:0/1372087050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd9741061f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd964009820 tx=0x7fd96402a270 comp rx=0 tx=0).stop 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.280+0000 7fd9789a2640 1 -- 172.21.15.64:0/1372087050 shutdown_connections 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.280+0000 7fd9789a2640 1 --2- 172.21.15.64:0/1372087050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd9741061f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.280+0000 7fd9789a2640 1 -- 172.21.15.64:0/1372087050 >> 172.21.15.64:0/1372087050 conn(0x7fd974101b60 msgr2=0x7fd974103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.280+0000 7fd9789a2640 1 -- 172.21.15.64:0/1372087050 shutdown_connections 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.280+0000 7fd9789a2640 1 -- 172.21.15.64:0/1372087050 wait complete. 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd9789a2640 1 Processor -- start 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd9789a2640 1 -- start start 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd9789a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd97419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd9789a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd974107640 con 0x7fd974105df0 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd97259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd97419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd97259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd97419d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40016/0 (socket says 172.21.15.64:40016) 2024-08-19T01:24:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd97259c640 1 -- 172.21.15.64:0/345321458 learned_addr learned my addr 172.21.15.64:0/345321458 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd97259c640 1 -- 172.21.15.64:0/345321458 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97419db80 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.281+0000 7fd97259c640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd97419d640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd964031000 tx=0x7fd9640326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.282+0000 7fd9637fe640 1 -- 172.21.15.64:0/345321458 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964042070 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.282+0000 7fd9637fe640 1 -- 172.21.15.64:0/345321458 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd964032be0 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.282+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97419de10 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.282+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9741a0b00 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.282+0000 7fd9637fe640 1 -- 172.21.15.64:0/345321458 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964037040 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.283+0000 7fd9637fe640 1 -- 172.21.15.64:0/345321458 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fd964045070 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.283+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9540052c0 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.283+0000 7fd9637fe640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fd95c038fc0 0x7fd95c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.283+0000 7fd9637fe640 1 -- 172.21.15.64:0/345321458 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd96406d020 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.283+0000 7fd971d9b640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fd95c038fc0 0x7fd95c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.284+0000 7fd971d9b640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fd95c038fc0 0x7fd95c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9680098c0 tx=0x7fd968006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.290+0000 7fd9637fe640 1 -- 172.21.15.64:0/345321458 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd964045360 con 0x7fd974105df0 2024-08-19T01:24:23.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.491+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd954000bf0 con 0x7fd95c038fc0 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.495+0000 7fd9637fe640 1 -- 172.21.15.64:0/345321458 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd954000bf0 con 0x7fd95c038fc0 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fd95c038fc0 msgr2=0x7fd95c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fd95c038fc0 0x7fd95c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9680098c0 tx=0x7fd968006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 msgr2=0x7fd97419d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd97419d640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd964031000 tx=0x7fd9640326e0 comp rx=0 tx=0).stop 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 shutdown_connections 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7fd95c038fc0 0x7fd95c03b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 --2- 172.21.15.64:0/345321458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd974105df0 0x7fd97419d640 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.497+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 >> 172.21.15.64:0/345321458 conn(0x7fd974101b60 msgr2=0x7fd9741942a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.498+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 shutdown_connections 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:22.498+0000 7fd9789a2640 1 -- 172.21.15.64:0/345321458 wait complete. 2024-08-19T01:24:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mgr service... 2024-08-19T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:23 smithi064 ceph-mon[18515]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:23 smithi064 ceph-mon[18515]: Saving service mon spec with placement count:5 2024-08-19T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:23 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.687+0000 7f045b447640 1 Processor -- start 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.687+0000 7f045b447640 1 -- start start 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.687+0000 7f045b447640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045410a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.687+0000 7f045b447640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f045407bda0 con 0x7f0454107f80 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.688+0000 7f04591e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045410a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.688+0000 7f04591e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045410a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40020/0 (socket says 172.21.15.64:40020) 2024-08-19T01:24:24.411 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.688+0000 7f04591e3640 1 -- 172.21.15.64:0/462481967 learned_addr learned my addr 172.21.15.64:0/462481967 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.688+0000 7f04591e3640 1 -- 172.21.15.64:0/462481967 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045410a8b0 con 0x7f0454107f80 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f04591e3640 1 --2- 172.21.15.64:0/462481967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045410a370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f044c009820 tx=0x7f044c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa673a2f34b90e3 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f0453fff640 1 -- 172.21.15.64:0/462481967 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f044c037070 con 0x7f0454107f80 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f0453fff640 1 -- 172.21.15.64:0/462481967 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f044c032900 con 0x7f0454107f80 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f0453fff640 1 -- 172.21.15.64:0/462481967 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f044c030850 con 0x7f0454107f80 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f045b447640 1 -- 172.21.15.64:0/462481967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 msgr2=0x7f045410a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f045b447640 1 --2- 172.21.15.64:0/462481967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045410a370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f044c009820 tx=0x7f044c02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f045b447640 1 -- 172.21.15.64:0/462481967 shutdown_connections 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f045b447640 1 --2- 172.21.15.64:0/462481967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045410a370 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.689+0000 7f045b447640 1 -- 172.21.15.64:0/462481967 >> 172.21.15.64:0/462481967 conn(0x7f0454101b60 msgr2=0x7f0454103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.690+0000 7f045b447640 1 -- 172.21.15.64:0/462481967 shutdown_connections 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.690+0000 7f045b447640 1 -- 172.21.15.64:0/462481967 wait complete. 2024-08-19T01:24:24.412 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.690+0000 7f045b447640 1 Processor -- start 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.690+0000 7f045b447640 1 -- start start 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.690+0000 7f045b447640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.690+0000 7f045b447640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f045410b830 con 0x7f0454107f80 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.690+0000 7f04591e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f04591e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045419b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40026/0 (socket says 172.21.15.64:40026) 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f04591e3640 1 -- 172.21.15.64:0/896814432 learned_addr learned my addr 172.21.15.64:0/896814432 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f04591e3640 1 -- 172.21.15.64:0/896814432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045419b940 con 0x7f0454107f80 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f04591e3640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045419b400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f044c032ce0 tx=0x7f044c032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f04527fc640 1 -- 172.21.15.64:0/896814432 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f044c042070 con 0x7f0454107f80 2024-08-19T01:24:24.413 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f04527fc640 1 -- 172.21.15.64:0/896814432 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f044c0308c0 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045419bbd0 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f04527fc640 1 -- 172.21.15.64:0/896814432 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f044c037040 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.691+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045419e8c0 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.692+0000 7f04527fc640 1 -- 172.21.15.64:0/896814432 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f044c045070 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.692+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04541083e0 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.692+0000 7f04527fc640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f043c038fc0 0x7f043c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.692+0000 7f04527fc640 1 -- 172.21.15.64:0/896814432 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f044c06bfd0 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.693+0000 7f04589e2640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f043c038fc0 0x7f043c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.693+0000 7f04589e2640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f043c038fc0 0x7f043c03b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f04480098c0 tx=0x7f0448006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.697+0000 7f04527fc640 1 -- 172.21.15.64:0/896814432 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f044c04b170 con 0x7f0454107f80 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.887+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0454102dd0 con 0x7f043c038fc0 2024-08-19T01:24:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.894+0000 7f04527fc640 1 -- 172.21.15.64:0/896814432 <== mgr.14118 v2:172.21.15.64:6800/2403632526 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0454102dd0 con 0x7f043c038fc0 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.896+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f043c038fc0 msgr2=0x7f043c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.896+0000 7f045b447640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f043c038fc0 0x7f043c03b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f04480098c0 tx=0x7f0448006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.896+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 msgr2=0x7f045419b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.896+0000 7f045b447640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045419b400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f044c032ce0 tx=0x7f044c032d10 comp rx=0 tx=0).stop 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.896+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 shutdown_connections 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.896+0000 7f045b447640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f043c038fc0 0x7f043c03b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.896+0000 7f045b447640 1 --2- 172.21.15.64:0/896814432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0454107f80 0x7f045419b400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.897+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 >> 172.21.15.64:0/896814432 conn(0x7f0454101b60 msgr2=0x7f04541026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.897+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 shutdown_connections 2024-08-19T01:24:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:23.897+0000 7f045b447640 1 -- 172.21.15.64:0/896814432 wait complete. 2024-08-19T01:24:25.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:24 smithi064 ceph-mon[18515]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:25.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:24 smithi064 ceph-mon[18515]: Saving service mgr spec with placement count:2 2024-08-19T01:24:25.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:24 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:26.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:26 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2015066482' entity='client.admin' 2024-08-19T01:24:26.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:26 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:26.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.827+0000 7f75e66bd640 1 Processor -- start 2024-08-19T01:24:26.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.827+0000 7f75e66bd640 1 -- start start 2024-08-19T01:24:26.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.827+0000 7f75e66bd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e010a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:26.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.827+0000 7f75e66bd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75e007be70 con 0x7f75e0107e90 2024-08-19T01:24:26.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.828+0000 7f75dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e010a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:26.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.828+0000 7f75dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e010a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40030/0 (socket says 172.21.15.64:40030) 2024-08-19T01:24:26.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.828+0000 7f75dffff640 1 -- 172.21.15.64:0/4170656545 learned_addr learned my addr 172.21.15.64:0/4170656545 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:26.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.828+0000 7f75dffff640 1 -- 172.21.15.64:0/4170656545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e010a7c0 con 0x7f75e0107e90 2024-08-19T01:24:26.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.828+0000 7f75dffff640 1 --2- 172.21.15.64:0/4170656545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e010a280 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009820 tx=0x7f75d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8faca798e1ae1c25 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:26.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75deffd640 1 -- 172.21.15.64:0/4170656545 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75d0037070 con 0x7f75e0107e90 2024-08-19T01:24:26.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75deffd640 1 -- 172.21.15.64:0/4170656545 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75d0032900 con 0x7f75e0107e90 2024-08-19T01:24:26.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75deffd640 1 -- 172.21.15.64:0/4170656545 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75d0030850 con 0x7f75e0107e90 2024-08-19T01:24:26.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75e66bd640 1 -- 172.21.15.64:0/4170656545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 msgr2=0x7f75e010a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:26.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75e66bd640 1 --2- 172.21.15.64:0/4170656545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e010a280 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009820 tx=0x7f75d002a270 comp rx=0 tx=0).stop 2024-08-19T01:24:26.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75e66bd640 1 -- 172.21.15.64:0/4170656545 shutdown_connections 2024-08-19T01:24:26.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75e66bd640 1 --2- 172.21.15.64:0/4170656545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e010a280 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:26.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75e66bd640 1 -- 172.21.15.64:0/4170656545 >> 172.21.15.64:0/4170656545 conn(0x7f75e0101b10 msgr2=0x7f75e0103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:26.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75e66bd640 1 -- 172.21.15.64:0/4170656545 shutdown_connections 2024-08-19T01:24:26.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.829+0000 7f75e66bd640 1 -- 172.21.15.64:0/4170656545 wait complete. 2024-08-19T01:24:26.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.830+0000 7f75e66bd640 1 Processor -- start 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.830+0000 7f75e66bd640 1 -- start start 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.830+0000 7f75e66bd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.830+0000 7f75e66bd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75e010b860 con 0x7f75e0107e90 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.830+0000 7f75dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.830+0000 7f75dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e019d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40042/0 (socket says 172.21.15.64:40042) 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.830+0000 7f75dffff640 1 -- 172.21.15.64:0/2015066482 learned_addr learned my addr 172.21.15.64:0/2015066482 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.831+0000 7f75dffff640 1 -- 172.21.15.64:0/2015066482 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e019db50 con 0x7f75e0107e90 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.831+0000 7f75dffff640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e019d610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75d00097f0 tx=0x7f75d0030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.831+0000 7f75dd7fa640 1 -- 172.21.15.64:0/2015066482 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75d0037040 con 0x7f75e0107e90 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.831+0000 7f75dd7fa640 1 -- 172.21.15.64:0/2015066482 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75d0040070 con 0x7f75e0107e90 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.831+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e019dde0 con 0x7f75e0107e90 2024-08-19T01:24:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.831+0000 7f75dd7fa640 1 -- 172.21.15.64:0/2015066482 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75d003a390 con 0x7f75e0107e90 2024-08-19T01:24:26.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.831+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e01a0ad0 con 0x7f75e0107e90 2024-08-19T01:24:26.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.832+0000 7f75dd7fa640 1 -- 172.21.15.64:0/2015066482 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f75d003a5e0 con 0x7f75e0107e90 2024-08-19T01:24:26.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.832+0000 7f75dd7fa640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f75c8038fc0 0x7f75c803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:26.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.832+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e01082f0 con 0x7f75e0107e90 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.832+0000 7f75dd7fa640 1 -- 172.21.15.64:0/2015066482 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f75d006c750 con 0x7f75e0107e90 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.832+0000 7f75df7fe640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f75c8038fc0 0x7f75c803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.833+0000 7f75df7fe640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f75c8038fc0 0x7f75c803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f75d40098c0 tx=0x7f75d4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:24.837+0000 7f75dd7fa640 1 -- 172.21.15.64:0/2015066482 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f75d002fd60 con 0x7f75e0107e90 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.017+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f75e01a1190 con 0x7f75e0107e90 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.019+0000 7f75dd7fa640 1 -- 172.21.15.64:0/2015066482 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f75d003a8d0 con 0x7f75e0107e90 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f75c8038fc0 msgr2=0x7f75c803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f75c8038fc0 0x7f75c803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f75d40098c0 tx=0x7f75d4006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 msgr2=0x7f75e019d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e019d610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75d00097f0 tx=0x7f75d0030d80 comp rx=0 tx=0).stop 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 shutdown_connections 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f75c8038fc0 0x7f75c803b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:26.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 --2- 172.21.15.64:0/2015066482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75e0107e90 0x7f75e019d610 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:26.086 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 >> 172.21.15.64:0/2015066482 conn(0x7f75e0101b10 msgr2=0x7f75e0102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:26.086 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 shutdown_connections 2024-08-19T01:24:26.086 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:25.022+0000 7f75e66bd640 1 -- 172.21.15.64:0/2015066482 wait complete. 2024-08-19T01:24:27.252 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.471+0000 7f58cbcb3640 1 Processor -- start 2024-08-19T01:24:27.252 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.471+0000 7f58cbcb3640 1 -- start start 2024-08-19T01:24:27.252 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.471+0000 7f58cbcb3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c410a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:27.252 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.471+0000 7f58cbcb3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f58c407be70 con 0x7f58c4107e90 2024-08-19T01:24:27.252 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.471+0000 7f58c9a4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c410a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.472+0000 7f58c9a4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c410a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40048/0 (socket says 172.21.15.64:40048) 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.472+0000 7f58c9a4f640 1 -- 172.21.15.64:0/2687410594 learned_addr learned my addr 172.21.15.64:0/2687410594 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.472+0000 7f58c9a4f640 1 -- 172.21.15.64:0/2687410594 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c410a7c0 con 0x7f58c4107e90 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.472+0000 7f58c9a4f640 1 --2- 172.21.15.64:0/2687410594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c410a280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009820 tx=0x7f58c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff354bb78b2ae31b server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58c8a4d640 1 -- 172.21.15.64:0/2687410594 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0037070 con 0x7f58c4107e90 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58c8a4d640 1 -- 172.21.15.64:0/2687410594 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58c0032920 con 0x7f58c4107e90 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58c8a4d640 1 -- 172.21.15.64:0/2687410594 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0030810 con 0x7f58c4107e90 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58cbcb3640 1 -- 172.21.15.64:0/2687410594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 msgr2=0x7f58c410a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58cbcb3640 1 --2- 172.21.15.64:0/2687410594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c410a280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009820 tx=0x7f58c002a270 comp rx=0 tx=0).stop 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58cbcb3640 1 -- 172.21.15.64:0/2687410594 shutdown_connections 2024-08-19T01:24:27.253 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58cbcb3640 1 --2- 172.21.15.64:0/2687410594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c410a280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.473+0000 7f58cbcb3640 1 -- 172.21.15.64:0/2687410594 >> 172.21.15.64:0/2687410594 conn(0x7f58c4101b10 msgr2=0x7f58c4103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.474+0000 7f58cbcb3640 1 -- 172.21.15.64:0/2687410594 shutdown_connections 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.474+0000 7f58cbcb3640 1 -- 172.21.15.64:0/2687410594 wait complete. 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.474+0000 7f58cbcb3640 1 Processor -- start 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.474+0000 7f58cbcb3640 1 -- start start 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.474+0000 7f58cbcb3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.474+0000 7f58cbcb3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f58c410b8a0 con 0x7f58c4107e90 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.474+0000 7f58c9a4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58c9a4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c419b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40050/0 (socket says 172.21.15.64:40050) 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58c9a4f640 1 -- 172.21.15.64:0/512185568 learned_addr learned my addr 172.21.15.64:0/512185568 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58c9a4f640 1 -- 172.21.15.64:0/512185568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c419b9a0 con 0x7f58c4107e90 2024-08-19T01:24:27.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58c9a4f640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c419b460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58c00097f0 tx=0x7f58c00327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58baffd640 1 -- 172.21.15.64:0/512185568 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0037040 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c419bc30 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58baffd640 1 -- 172.21.15.64:0/512185568 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58c0030cc0 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58baffd640 1 -- 172.21.15.64:0/512185568 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c003a6b0 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.475+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c419e920 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.476+0000 7f58baffd640 1 -- 172.21.15.64:0/512185568 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f58c003a8a0 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.476+0000 7f58baffd640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f58b0038f70 0x7f58b003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.477+0000 7f58baffd640 1 -- 172.21.15.64:0/512185568 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f58c006c670 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.477+0000 7f58c924e640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f58b0038f70 0x7f58b003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.477+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c41082f0 con 0x7f58c4107e90 2024-08-19T01:24:27.255 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.477+0000 7f58c924e640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f58b0038f70 0x7f58b003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f58b40098c0 tx=0x7f58b4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.482+0000 7f58baffd640 1 -- 172.21.15.64:0/512185568 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f58c0044920 con 0x7f58c4107e90 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.658+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f58c419ee70 con 0x7f58c4107e90 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.663+0000 7f58baffd640 1 -- 172.21.15.64:0/512185568 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f58c0044400 con 0x7f58c4107e90 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f58b0038f70 msgr2=0x7f58b003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f58b0038f70 0x7f58b003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f58b40098c0 tx=0x7f58b4006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 msgr2=0x7f58c419b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c419b460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58c00097f0 tx=0x7f58c00327b0 comp rx=0 tx=0).stop 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 shutdown_connections 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f58b0038f70 0x7f58b003b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 --2- 172.21.15.64:0/512185568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58c4107e90 0x7f58c419b460 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 >> 172.21.15.64:0/512185568 conn(0x7f58c4101b10 msgr2=0x7f58c4102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 shutdown_connections 2024-08-19T01:24:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:26.666+0000 7f58cbcb3640 1 -- 172.21.15.64:0/512185568 wait complete. 2024-08-19T01:24:27.257 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling the dashboard module... 2024-08-19T01:24:27.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:27 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:27.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:27 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/512185568' entity='client.admin' 2024-08-19T01:24:28.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:28 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3402995868' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-08-19T01:24:28.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:28 smithi064 ceph-mon[18515]: from='mgr.14118 172.21.15.64:0/2388279968' entity='mgr.x' 2024-08-19T01:24:28.843 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:28 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ignoring --setuser ceph since I am not root 2024-08-19T01:24:28.844 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:28 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ignoring --setgroup ceph since I am not root 2024-08-19T01:24:28.844 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:28 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:28.771+0000 7f713bbb5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.624+0000 7f2b9e158640 1 Processor -- start 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.624+0000 7f2b9e158640 1 -- start start 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.624+0000 7f2b9e158640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b98106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.624+0000 7f2b9e158640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98106d00 con 0x7f2b98106330 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.625+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b98106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.625+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b98106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40062/0 (socket says 172.21.15.64:40062) 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.625+0000 7f2b977fe640 1 -- 172.21.15.64:0/2465731784 learned_addr learned my addr 172.21.15.64:0/2465731784 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.625+0000 7f2b977fe640 1 -- 172.21.15.64:0/2465731784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b98107530 con 0x7f2b98106330 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b977fe640 1 --2- 172.21.15.64:0/2465731784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b98106730 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009820 tx=0x7f2b8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6c471c2bb29e18 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b967fc640 1 -- 172.21.15.64:0/2465731784 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b88037070 con 0x7f2b98106330 2024-08-19T01:24:29.095 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b967fc640 1 -- 172.21.15.64:0/2465731784 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b88032920 con 0x7f2b98106330 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b967fc640 1 -- 172.21.15.64:0/2465731784 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b88030810 con 0x7f2b98106330 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b9e158640 1 -- 172.21.15.64:0/2465731784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 msgr2=0x7f2b98106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b9e158640 1 --2- 172.21.15.64:0/2465731784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b98106730 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009820 tx=0x7f2b8802a270 comp rx=0 tx=0).stop 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b9e158640 1 -- 172.21.15.64:0/2465731784 shutdown_connections 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b9e158640 1 --2- 172.21.15.64:0/2465731784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b98106730 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b9e158640 1 -- 172.21.15.64:0/2465731784 >> 172.21.15.64:0/2465731784 conn(0x7f2b98101ac0 msgr2=0x7f2b98103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.626+0000 7f2b9e158640 1 -- 172.21.15.64:0/2465731784 shutdown_connections 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.627+0000 7f2b9e158640 1 -- 172.21.15.64:0/2465731784 wait complete. 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.627+0000 7f2b9e158640 1 Processor -- start 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.627+0000 7f2b9e158640 1 -- start start 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.627+0000 7f2b9e158640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b9819b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.627+0000 7f2b9e158640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9810a8e0 con 0x7f2b98106330 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.627+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b9819b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:29.096 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b9819b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40070/0 (socket says 172.21.15.64:40070) 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b977fe640 1 -- 172.21.15.64:0/3402995868 learned_addr learned my addr 172.21.15.64:0/3402995868 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b977fe640 1 -- 172.21.15.64:0/3402995868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9819b9b0 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b977fe640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b9819b470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b880097f0 tx=0x7f2b880327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b88037040 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b88030cc0 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9819bc40 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b8803a640 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.628+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b9819e930 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.629+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2b8803a8f0 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.629+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b9810aa60 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.629+0000 7f2b94ff9640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f2b80038fc0 0x7f2b8003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.629+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2b8806c560 con 0x7f2b98106330 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.629+0000 7f2b96ffd640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f2b80038fc0 0x7f2b8003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.630+0000 7f2b96ffd640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f2b80038fc0 0x7f2b8003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098c0 tx=0x7f2b8c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:29.097 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.634+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f2b880328c0 con 0x7f2b98106330 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:27.902+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f2b9802d7d0 con 0x7f2b98106330 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.526+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f2b88031990 con 0x7f2b98106330 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.528+0000 7f2b94ff9640 1 -- 172.21.15.64:0/3402995868 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f2b8802fd40 con 0x7f2b98106330 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f2b80038fc0 msgr2=0x7f2b8003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f2b80038fc0 0x7f2b8003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098c0 tx=0x7f2b8c006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 msgr2=0x7f2b9819b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b9819b470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b880097f0 tx=0x7f2b880327b0 comp rx=0 tx=0).stop 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 shutdown_connections 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f2b80038fc0 0x7f2b8003b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 --2- 172.21.15.64:0/3402995868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b98106330 0x7f2b9819b470 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.529+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 >> 172.21.15.64:0/3402995868 conn(0x7f2b98101ac0 msgr2=0x7f2b98102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.530+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 shutdown_connections 2024-08-19T01:24:29.098 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:28.530+0000 7f2b9e158640 1 -- 172.21.15.64:0/3402995868 wait complete. 2024-08-19T01:24:29.161 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:28 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:28.841+0000 7f713bbb5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:24:29.161 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:28 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:28.990+0000 7f713bbb5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:24:29.161 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:29.158+0000 7f713bbb5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:24:29.415 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:29.310+0000 7f713bbb5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:24:29.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3402995868' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-08-19T01:24:29.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-mon[18515]: mgrmap e9: x(active, since 18s) 2024-08-19T01:24:29.700 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:29.443+0000 7f713bbb5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:24:29.701 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:29.575+0000 7f713bbb5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:24:29.701 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:29.634+0000 7f713bbb5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:24:30.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:29.951+0000 7f713bbb5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:24:30.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.015+0000 7f713bbb5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:24:30.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.086+0000 7f713bbb5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:24:30.277 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-08-19T01:24:30.277 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-08-19T01:24:30.277 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-19T01:24:30.277 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-08-19T01:24:30.277 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-19T01:24:30.277 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.492+0000 7ffa89fe2640 1 Processor -- start 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.492+0000 7ffa89fe2640 1 -- start start 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.492+0000 7ffa89fe2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8410a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.492+0000 7ffa89fe2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8407be70 con 0x7ffa84107e90 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.492+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8410a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.492+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8410a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33120/0 (socket says 172.21.15.64:33120) 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.492+0000 7ffa837fe640 1 -- 172.21.15.64:0/1852510050 learned_addr learned my addr 172.21.15.64:0/1852510050 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.493+0000 7ffa837fe640 1 -- 172.21.15.64:0/1852510050 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8410a7c0 con 0x7ffa84107e90 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.493+0000 7ffa837fe640 1 --2- 172.21.15.64:0/1852510050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8410a280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009820 tx=0x7ffa7402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39491bec786cd5ec server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.493+0000 7ffa827fc640 1 -- 172.21.15.64:0/1852510050 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa74037070 con 0x7ffa84107e90 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.493+0000 7ffa827fc640 1 -- 172.21.15.64:0/1852510050 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa74032920 con 0x7ffa84107e90 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.493+0000 7ffa827fc640 1 -- 172.21.15.64:0/1852510050 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa74030810 con 0x7ffa84107e90 2024-08-19T01:24:30.278 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.494+0000 7ffa89fe2640 1 -- 172.21.15.64:0/1852510050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 msgr2=0x7ffa8410a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.494+0000 7ffa89fe2640 1 --2- 172.21.15.64:0/1852510050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8410a280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009820 tx=0x7ffa7402a270 comp rx=0 tx=0).stop 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.494+0000 7ffa89fe2640 1 -- 172.21.15.64:0/1852510050 shutdown_connections 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.494+0000 7ffa89fe2640 1 --2- 172.21.15.64:0/1852510050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8410a280 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.494+0000 7ffa89fe2640 1 -- 172.21.15.64:0/1852510050 >> 172.21.15.64:0/1852510050 conn(0x7ffa84101b10 msgr2=0x7ffa84103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.494+0000 7ffa89fe2640 1 -- 172.21.15.64:0/1852510050 shutdown_connections 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.494+0000 7ffa89fe2640 1 -- 172.21.15.64:0/1852510050 wait complete. 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.495+0000 7ffa89fe2640 1 Processor -- start 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.495+0000 7ffa89fe2640 1 -- start start 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.495+0000 7ffa89fe2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.495+0000 7ffa89fe2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8410b8a0 con 0x7ffa84107e90 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.495+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.495+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8419d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33128/0 (socket says 172.21.15.64:33128) 2024-08-19T01:24:30.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.495+0000 7ffa837fe640 1 -- 172.21.15.64:0/3578041019 learned_addr learned my addr 172.21.15.64:0/3578041019 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.496+0000 7ffa837fe640 1 -- 172.21.15.64:0/3578041019 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8419dbf0 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.496+0000 7ffa837fe640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8419d6b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffa740097f0 tx=0x7ffa740327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.496+0000 7ffa80ff9640 1 -- 172.21.15.64:0/3578041019 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa74037040 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.496+0000 7ffa80ff9640 1 -- 172.21.15.64:0/3578041019 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa74030cc0 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.496+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa8419de80 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.496+0000 7ffa80ff9640 1 -- 172.21.15.64:0/3578041019 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa7403a6b0 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.496+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa841a0b70 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.497+0000 7ffa80ff9640 1 -- 172.21.15.64:0/3578041019 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7ffa7403a850 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.497+0000 7ffa80ff9640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 0x7ffa6c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.497+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa841082f0 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.497+0000 7ffa82ffd640 1 -- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 msgr2=0x7ffa6c03b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2403632526 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.497+0000 7ffa80ff9640 1 -- 172.21.15.64:0/3578041019 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ffa7406c4f0 con 0x7ffa84107e90 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.497+0000 7ffa82ffd640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 0x7ffa6c03b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:24:30.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.502+0000 7ffa80ff9640 1 -- 172.21.15.64:0/3578041019 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ffa7402de20 con 0x7ffa84107e90 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.698+0000 7ffa82ffd640 1 -- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 msgr2=0x7ffa6c03b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2403632526 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.698+0000 7ffa82ffd640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 0x7ffa6c03b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.726+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7ffa841a1160 con 0x7ffa84107e90 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.727+0000 7ffa80ff9640 1 -- 172.21.15.64:0/3578041019 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7ffa74044920 con 0x7ffa84107e90 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 msgr2=0x7ffa6c03b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 0x7ffa6c03b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 msgr2=0x7ffa8419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8419d6b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffa740097f0 tx=0x7ffa740327b0 comp rx=0 tx=0).stop 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 shutdown_connections 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7ffa6c039020 0x7ffa6c03b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 --2- 172.21.15.64:0/3578041019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa84107e90 0x7ffa8419d6b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 >> 172.21.15.64:0/3578041019 conn(0x7ffa84101b10 msgr2=0x7ffa84102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:30.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 shutdown_connections 2024-08-19T01:24:30.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:29.729+0000 7ffa89fe2640 1 -- 172.21.15.64:0/3578041019 wait complete. 2024-08-19T01:24:30.282 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-08-19T01:24:30.282 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 9... 2024-08-19T01:24:30.530 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.238+0000 7f713bbb5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:24:30.530 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.308+0000 7f713bbb5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:24:30.530 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.377+0000 7f713bbb5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:24:30.780 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3578041019' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-19T01:24:30.780 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.576+0000 7f713bbb5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:24:30.780 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.642+0000 7f713bbb5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:24:30.780 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.706+0000 7f713bbb5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:24:30.780 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:30.778+0000 7f713bbb5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:24:31.068 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:31 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:31.004+0000 7f713bbb5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:24:31.068 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:31 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:31.066+0000 7f713bbb5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:24:31.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:31 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:31.604+0000 7f713bbb5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:24:32.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:31 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:31.925+0000 7f713bbb5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:24:32.738 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:32 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:32.354+0000 7f713bbb5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:24:32.738 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:32 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:32.412+0000 7f713bbb5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:24:32.738 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:32 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:32.474+0000 7f713bbb5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:24:33.862 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: Active manager daemon x restarted 2024-08-19T01:24:33.862 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: Activating manager daemon x 2024-08-19T01:24:33.862 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: osdmap e3: 0 total, 0 up, 0 in 2024-08-19T01:24:33.862 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: mgrmap e10: x(active, starting, since 0.00347109s) 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: Manager daemon x is now available 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:24:33.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:24:35.509 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-08-19T01:24:35.509 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-08-19T01:24:35.509 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-19T01:24:35.509 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-08-19T01:24:35.509 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.628+0000 7f22db24b640 1 Processor -- start 2024-08-19T01:24:35.510 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.628+0000 7f22db24b640 1 -- start start 2024-08-19T01:24:35.510 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.628+0000 7f22db24b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d4106720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:35.510 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.628+0000 7f22db24b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4106cf0 con 0x7f22d4106320 2024-08-19T01:24:35.510 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.629+0000 7f22d8fe7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:35.510 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.629+0000 7f22d8fe7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33136/0 (socket says 172.21.15.64:33136) 2024-08-19T01:24:35.510 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.629+0000 7f22d8fe7640 1 -- 172.21.15.64:0/692233132 learned_addr learned my addr 172.21.15.64:0/692233132 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:35.510 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.629+0000 7f22d8fe7640 1 -- 172.21.15.64:0/692233132 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d4107520 con 0x7f22d4106320 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.630+0000 7f22d8fe7640 1 --2- 172.21.15.64:0/692233132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d4106720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009820 tx=0x7f22c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca02a7a0c08ef598 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.630+0000 7f22d37fe640 1 -- 172.21.15.64:0/692233132 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22c8037070 con 0x7f22d4106320 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.630+0000 7f22d37fe640 1 -- 172.21.15.64:0/692233132 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c8032920 con 0x7f22d4106320 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.630+0000 7f22d37fe640 1 -- 172.21.15.64:0/692233132 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22c8030810 con 0x7f22d4106320 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 -- 172.21.15.64:0/692233132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 msgr2=0x7f22d4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 --2- 172.21.15.64:0/692233132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d4106720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009820 tx=0x7f22c802a270 comp rx=0 tx=0).stop 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 -- 172.21.15.64:0/692233132 shutdown_connections 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 --2- 172.21.15.64:0/692233132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d4106720 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 -- 172.21.15.64:0/692233132 >> 172.21.15.64:0/692233132 conn(0x7f22d4101b10 msgr2=0x7f22d4103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 -- 172.21.15.64:0/692233132 shutdown_connections 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 -- 172.21.15.64:0/692233132 wait complete. 2024-08-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 Processor -- start 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 -- start start 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.631+0000 7f22db24b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d407ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22db24b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f22d410a8d0 con 0x7f22d4106320 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d8fe7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d407ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d8fe7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d407ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33142/0 (socket says 172.21.15.64:33142) 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d8fe7640 1 -- 172.21.15.64:0/4259172283 learned_addr learned my addr 172.21.15.64:0/4259172283 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d8fe7640 1 -- 172.21.15.64:0/4259172283 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d419e200 con 0x7f22d4106320 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d8fe7640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d407ba10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f22c8030560 tx=0x7f22c8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22c8037040 con 0x7f22d4106320 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d419e3d0 con 0x7f22d4106320 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c8040070 con 0x7f22d4106320 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22c803a740 con 0x7f22d4106320 2024-08-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.632+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d419e770 con 0x7f22d4106320 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.634+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f22c803a8e0 con 0x7f22d4106320 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.634+0000 7f22d1ffb640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 0x7f22c403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.634+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 --> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f22c403bbf0 con 0x7f22c4039020 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.634+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f22c806c5f0 con 0x7f22d4106320 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.634+0000 7f22d3fff640 1 -- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 msgr2=0x7f22c403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2403632526 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.634+0000 7f22d3fff640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 0x7f22c403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.835+0000 7f22d3fff640 1 -- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 msgr2=0x7f22c403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2403632526 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:30.835+0000 7f22d3fff640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 0x7f22c403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:31.236+0000 7f22d3fff640 1 -- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 msgr2=0x7f22c403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2403632526 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:31.236+0000 7f22d3fff640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 0x7f22c403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:32.037+0000 7f22d3fff640 1 -- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 msgr2=0x7f22c403b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2403632526 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:32.037+0000 7f22d3fff640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 0x7f22c403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:33.599+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f22c806b0a0 con 0x7f22d4106320 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:33.599+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 msgr2=0x7f22c403b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:33.599+0000 7f22d1ffb640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 0x7f22c403b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.600+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f22c803b420 con 0x7f22d4106320 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.601+0000 7f22d1ffb640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f22c403cae0 0x7f22c403eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.601+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f22c403bbf0 con 0x7f22c403cae0 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.601+0000 7f22d3fff640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f22c403cae0 0x7f22c403eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.602+0000 7f22d3fff640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f22c403cae0 0x7f22c403eed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f22c0003980 tx=0x7f22c00092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.602+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f22c403bbf0 con 0x7f22c403cae0 2024-08-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f22d410aa50 con 0x7f22c403cae0 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22d1ffb640 1 -- 172.21.15.64:0/4259172283 <== mgr.14150 v2:172.21.15.64:6800/2663741075 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f22d410aa50 con 0x7f22c403cae0 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f22c403cae0 msgr2=0x7f22c403eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f22c403cae0 0x7f22c403eed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f22c0003980 tx=0x7f22c00092b0 comp rx=0 tx=0).stop 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 msgr2=0x7f22d407ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d407ba10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f22c8030560 tx=0x7f22c8032670 comp rx=0 tx=0).stop 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 shutdown_connections 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f22c403cae0 0x7f22c403eed0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:6800/2403632526,v1:172.21.15.64:6801/2403632526] conn(0x7f22c4039020 0x7f22c403b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 --2- 172.21.15.64:0/4259172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22d4106320 0x7f22d407ba10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.615+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 >> 172.21.15.64:0/4259172283 conn(0x7f22d4101b10 msgr2=0x7f22d41026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.616+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 shutdown_connections 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:34.616+0000 7f22db24b640 1 -- 172.21.15.64:0/4259172283 wait complete. 2024-08-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 9 is available 2024-08-19T01:24:35.521 INFO:teuthology.orchestra.run.smithi064.stdout:Generating a dashboard self-signed certificate... 2024-08-19T01:24:35.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:35 smithi064 ceph-mon[18515]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-19T01:24:35.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:35 smithi064 ceph-mon[18515]: mgrmap e11: x(active, since 1.00589s) 2024-08-19T01:24:35.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:35 smithi064 ceph-mon[18515]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-19T01:24:35.764 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-mon[18515]: mgrmap e12: x(active, since 2s) 2024-08-19T01:24:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:36] ENGINE Bus STARTING 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:36] ENGINE Serving on https://172.21.15.64:7150 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:36] ENGINE Error in HTTPServer.serve 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._connections.run(self.expiration_interval) 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._run(expiration_interval) 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:24:36.986 INFO:journalctl@ceph.mon.a.smithi064.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: s = self.context.wrap_socket( 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: return self.sslsocket_class._create( 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: self.do_handshake() 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._sslobj.do_handshake() 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mon.a.smithi064.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:36.538+0000 7f70a6ebd640 -1 log_channel(cephadm) log [ERR] : [19/Aug/2024:01:24:36] ENGINE Error in HTTPServer.serve 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: Traceback (most recent call last): 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:24:36.987 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._connections.run(self.expiration_interval) 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._run(expiration_interval) 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: s = self.context.wrap_socket( 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: return self.sslsocket_class._create( 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self.do_handshake() 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._sslobj.do_handshake() 2024-08-19T01:24:36.988 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:24:36.989 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:24:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.971+0000 7f6c10805640 1 Processor -- start 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.971+0000 7f6c10805640 1 -- start start 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.972+0000 7f6c10805640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c1060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.972+0000 7f6c10805640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c07be30 con 0x7f6c0c105cc0 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.972+0000 7f6c0a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c1060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.972+0000 7f6c0a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c1060c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33218/0 (socket says 172.21.15.64:33218) 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.972+0000 7f6c0a59c640 1 -- 172.21.15.64:0/1170762793 learned_addr learned my addr 172.21.15.64:0/1170762793 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:37.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.972+0000 7f6c0a59c640 1 -- 172.21.15.64:0/1170762793 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c106600 con 0x7f6c0c105cc0 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.973+0000 7f6c0a59c640 1 --2- 172.21.15.64:0/1170762793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c1060c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009820 tx=0x7f6bfc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7b3e8956a23934 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c0959a640 1 -- 172.21.15.64:0/1170762793 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bfc037070 con 0x7f6c0c105cc0 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c0959a640 1 -- 172.21.15.64:0/1170762793 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bfc032920 con 0x7f6c0c105cc0 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c0959a640 1 -- 172.21.15.64:0/1170762793 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bfc030810 con 0x7f6c0c105cc0 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c10805640 1 -- 172.21.15.64:0/1170762793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 msgr2=0x7f6c0c1060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c10805640 1 --2- 172.21.15.64:0/1170762793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c1060c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009820 tx=0x7f6bfc02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c10805640 1 -- 172.21.15.64:0/1170762793 shutdown_connections 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c10805640 1 --2- 172.21.15.64:0/1170762793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c1060c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c10805640 1 -- 172.21.15.64:0/1170762793 >> 172.21.15.64:0/1170762793 conn(0x7f6c0c101b10 msgr2=0x7f6c0c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c10805640 1 -- 172.21.15.64:0/1170762793 shutdown_connections 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.974+0000 7f6c10805640 1 -- 172.21.15.64:0/1170762793 wait complete. 2024-08-19T01:24:37.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.975+0000 7f6c10805640 1 Processor -- start 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.975+0000 7f6c10805640 1 -- start start 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.975+0000 7f6c10805640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.975+0000 7f6c10805640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c107740 con 0x7f6c0c105cc0 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6c0a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6c0a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c19d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33220/0 (socket says 172.21.15.64:33220) 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6c0a59c640 1 -- 172.21.15.64:0/1375882866 learned_addr learned my addr 172.21.15.64:0/1375882866 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6c0a59c640 1 -- 172.21.15.64:0/1375882866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c19db70 con 0x7f6c0c105cc0 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6c0a59c640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c19d630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097f0 tx=0x7f6bfc0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6bfb7fe640 1 -- 172.21.15.64:0/1375882866 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bfc037040 con 0x7f6c0c105cc0 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c19de00 con 0x7f6c0c105cc0 2024-08-19T01:24:37.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6bfb7fe640 1 -- 172.21.15.64:0/1375882866 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bfc030cc0 con 0x7f6c0c105cc0 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.976+0000 7f6bfb7fe640 1 -- 172.21.15.64:0/1375882866 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bfc03a6b0 con 0x7f6c0c105cc0 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.977+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c1a0af0 con 0x7f6c0c105cc0 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.977+0000 7f6bfb7fe640 1 -- 172.21.15.64:0/1375882866 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f6bfc03a8a0 con 0x7f6c0c105cc0 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.978+0000 7f6bfb7fe640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6bf4038fc0 0x7f6bf403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.978+0000 7f6bfb7fe640 1 -- 172.21.15.64:0/1375882866 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6bfc071660 con 0x7f6c0c105cc0 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.978+0000 7f6c09d9b640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6bf4038fc0 0x7f6bf403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.978+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c1060c0 con 0x7f6c0c105cc0 2024-08-19T01:24:37.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.978+0000 7f6c09d9b640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6bf4038fc0 0x7f6bf403b480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c000098c0 tx=0x7f6c00006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:35.987+0000 7f6bfb7fe640 1 -- 172.21.15.64:0/1375882866 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6bfc039950 con 0x7f6c0c105cc0 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.189+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f6c0c062360 con 0x7f6bf4038fc0 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.262+0000 7f6bfb7fe640 1 -- 172.21.15.64:0/1375882866 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f6c0c062360 con 0x7f6bf4038fc0 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6bf4038fc0 msgr2=0x7f6bf403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6bf4038fc0 0x7f6bf403b480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c000098c0 tx=0x7f6c00006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 msgr2=0x7f6c0c19d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c19d630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097f0 tx=0x7f6bfc0327b0 comp rx=0 tx=0).stop 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 shutdown_connections 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6bf4038fc0 0x7f6bf403b480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 --2- 172.21.15.64:0/1375882866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c0c105cc0 0x7f6c0c19d630 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 >> 172.21.15.64:0/1375882866 conn(0x7f6c0c101b10 msgr2=0x7f6c0c191930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 shutdown_connections 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:36.264+0000 7f6c10805640 1 -- 172.21.15.64:0/1375882866 wait complete. 2024-08-19T01:24:37.285 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial admin user... 2024-08-19T01:24:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:37 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:36] ENGINE Serving on http://172.21.15.64:8765 2024-08-19T01:24:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:37 smithi064 ceph-mon[18515]: [19/Aug/2024:01:24:36] ENGINE Bus STARTED 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$IoaYWt.xT7gHDKLBz0wOUu3BYokRvVFs0eV.W/K1QN0aNGFIGFhFm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1724030678, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.107+0000 7f21d0930640 1 Processor -- start 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.107+0000 7f21d0930640 1 -- start start 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.107+0000 7f21d0930640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc106780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.107+0000 7f21d0930640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc106d50 con 0x7f21cc106380 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.107+0000 7f21ca59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.107+0000 7f21ca59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33232/0 (socket says 172.21.15.64:33232) 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.107+0000 7f21ca59c640 1 -- 172.21.15.64:0/1636456717 learned_addr learned my addr 172.21.15.64:0/1636456717 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.108+0000 7f21ca59c640 1 -- 172.21.15.64:0/1636456717 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc107580 con 0x7f21cc106380 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.108+0000 7f21ca59c640 1 --2- 172.21.15.64:0/1636456717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc106780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009820 tx=0x7f21bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8012638dee447e50 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21c959a640 1 -- 172.21.15.64:0/1636456717 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21bc037070 con 0x7f21cc106380 2024-08-19T01:24:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21c959a640 1 -- 172.21.15.64:0/1636456717 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21bc032920 con 0x7f21cc106380 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21c959a640 1 -- 172.21.15.64:0/1636456717 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21bc030810 con 0x7f21cc106380 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21d0930640 1 -- 172.21.15.64:0/1636456717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 msgr2=0x7f21cc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21d0930640 1 --2- 172.21.15.64:0/1636456717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc106780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009820 tx=0x7f21bc02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21d0930640 1 -- 172.21.15.64:0/1636456717 shutdown_connections 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21d0930640 1 --2- 172.21.15.64:0/1636456717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc106780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21d0930640 1 -- 172.21.15.64:0/1636456717 >> 172.21.15.64:0/1636456717 conn(0x7f21cc101b90 msgr2=0x7f21cc103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21d0930640 1 -- 172.21.15.64:0/1636456717 shutdown_connections 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.109+0000 7f21d0930640 1 -- 172.21.15.64:0/1636456717 wait complete. 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.110+0000 7f21d0930640 1 Processor -- start 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.110+0000 7f21d0930640 1 -- start start 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.110+0000 7f21d0930640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc19b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.110+0000 7f21d0930640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc10a930 con 0x7f21cc106380 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21ca59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc19b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21ca59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc19b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33248/0 (socket says 172.21.15.64:33248) 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21ca59c640 1 -- 172.21.15.64:0/3425993989 learned_addr learned my addr 172.21.15.64:0/3425993989 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21ca59c640 1 -- 172.21.15.64:0/3425993989 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc19ba80 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21ca59c640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc19b540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f21bc030560 tx=0x7f21bc032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21bb7fe640 1 -- 172.21.15.64:0/3425993989 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21bc037040 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21bb7fe640 1 -- 172.21.15.64:0/3425993989 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21bc040070 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21cc19bd10 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21bb7fe640 1 -- 172.21.15.64:0/3425993989 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21bc03a6d0 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.111+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21cc19ea00 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.112+0000 7f21bb7fe640 1 -- 172.21.15.64:0/3425993989 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f21bc03a870 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.113+0000 7f21bb7fe640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f21b4039020 0x7f21b403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.113+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21cc10aab0 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.113+0000 7f21bb7fe640 1 -- 172.21.15.64:0/3425993989 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f21bc06c780 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.113+0000 7f21c9d9b640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f21b4039020 0x7f21b403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.113+0000 7f21c9d9b640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f21b4039020 0x7f21b403b4e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21c00098c0 tx=0x7f21c0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.118+0000 7f21bb7fe640 1 -- 172.21.15.64:0/3425993989 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f21bc032920 con 0x7f21cc106380 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.307+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f21cc106800 con 0x7f21b4039020 2024-08-19T01:24:39.125 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.555+0000 7f21bb7fe640 1 -- 172.21.15.64:0/3425993989 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f21cc106800 con 0x7f21b4039020 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f21b4039020 msgr2=0x7f21b403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f21b4039020 0x7f21b403b4e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21c00098c0 tx=0x7f21c0006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 msgr2=0x7f21cc19b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc19b540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f21bc030560 tx=0x7f21bc032670 comp rx=0 tx=0).stop 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 shutdown_connections 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f21b4039020 0x7f21b403b4e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 --2- 172.21.15.64:0/3425993989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21cc106380 0x7f21cc19b540 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.557+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 >> 172.21.15.64:0/3425993989 conn(0x7f21cc101b90 msgr2=0x7f21cc102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:39.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.558+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 shutdown_connections 2024-08-19T01:24:39.130 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:38.558+0000 7f21d0930640 1 -- 172.21.15.64:0/3425993989 wait complete. 2024-08-19T01:24:39.130 INFO:teuthology.orchestra.run.smithi064.stdout:Fetching dashboard port number... 2024-08-19T01:24:39.660 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:39 smithi064 ceph-mon[18515]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:39.660 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:40.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 8443 2024-08-19T01:24:40.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.471+0000 7fa7ef254640 1 Processor -- start 2024-08-19T01:24:40.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.471+0000 7fa7ef254640 1 -- start start 2024-08-19T01:24:40.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.471+0000 7fa7ef254640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e8107910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:40.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.471+0000 7fa7ef254640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e8107ee0 con 0x7fa7e8107510 2024-08-19T01:24:40.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.472+0000 7fa7ecff0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e8107910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.472+0000 7fa7ecff0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e8107910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43302/0 (socket says 172.21.15.64:43302) 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.472+0000 7fa7ecff0640 1 -- 172.21.15.64:0/3820053818 learned_addr learned my addr 172.21.15.64:0/3820053818 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.472+0000 7fa7ecff0640 1 -- 172.21.15.64:0/3820053818 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e81086c0 con 0x7fa7e8107510 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.473+0000 7fa7ecff0640 1 --2- 172.21.15.64:0/3820053818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e8107910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009820 tx=0x7fa7dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89f940966e46167 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.473+0000 7fa7e77fe640 1 -- 172.21.15.64:0/3820053818 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7dc037070 con 0x7fa7e8107510 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.473+0000 7fa7e77fe640 1 -- 172.21.15.64:0/3820053818 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7dc032920 con 0x7fa7e8107510 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.473+0000 7fa7e77fe640 1 -- 172.21.15.64:0/3820053818 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7dc030810 con 0x7fa7e8107510 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 -- 172.21.15.64:0/3820053818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 msgr2=0x7fa7e8107910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 --2- 172.21.15.64:0/3820053818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e8107910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009820 tx=0x7fa7dc02a270 comp rx=0 tx=0).stop 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 -- 172.21.15.64:0/3820053818 shutdown_connections 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 --2- 172.21.15.64:0/3820053818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e8107910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 -- 172.21.15.64:0/3820053818 >> 172.21.15.64:0/3820053818 conn(0x7fa7e8102c60 msgr2=0x7fa7e81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 -- 172.21.15.64:0/3820053818 shutdown_connections 2024-08-19T01:24:40.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 -- 172.21.15.64:0/3820053818 wait complete. 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 Processor -- start 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 -- start start 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e819b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.474+0000 7fa7ef254640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e810ba70 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7ecff0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e819b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7ecff0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e819b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43312/0 (socket says 172.21.15.64:43312) 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7ecff0640 1 -- 172.21.15.64:0/3571307335 learned_addr learned my addr 172.21.15.64:0/3571307335 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7ecff0640 1 -- 172.21.15.64:0/3571307335 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e819ba70 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7ecff0640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e819b530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0097f0 tx=0x7fa7dc0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7e5ffb640 1 -- 172.21.15.64:0/3571307335 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7dc037040 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7e5ffb640 1 -- 172.21.15.64:0/3571307335 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7dc030c40 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e819bd00 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7e5ffb640 1 -- 172.21.15.64:0/3571307335 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7dc03a5b0 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.475+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e819e9f0 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.476+0000 7fa7e5ffb640 1 -- 172.21.15.64:0/3571307335 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fa7dc03a7a0 con 0x7fa7e8107510 2024-08-19T01:24:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.477+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e8107990 con 0x7fa7e8107510 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.477+0000 7fa7e5ffb640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa7d8039020 0x7fa7d803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.477+0000 7fa7e5ffb640 1 -- 172.21.15.64:0/3571307335 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa7dc06c5e0 con 0x7fa7e8107510 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.477+0000 7fa7e7fff640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa7d8039020 0x7fa7d803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.477+0000 7fa7e7fff640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa7d8039020 0x7fa7d803b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40098c0 tx=0x7fa7d4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.481+0000 7fa7e5ffb640 1 -- 172.21.15.64:0/3571307335 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa7dc031c00 con 0x7fa7e8107510 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.657+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fa7e819efe0 con 0x7fa7e8107510 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.658+0000 7fa7e5ffb640 1 -- 172.21.15.64:0/3571307335 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fa7dc044920 con 0x7fa7e8107510 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.659+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa7d8039020 msgr2=0x7fa7d803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.659+0000 7fa7ef254640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa7d8039020 0x7fa7d803b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40098c0 tx=0x7fa7d4006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.659+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 msgr2=0x7fa7e819b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.659+0000 7fa7ef254640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e819b530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0097f0 tx=0x7fa7dc0327b0 comp rx=0 tx=0).stop 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.660+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 shutdown_connections 2024-08-19T01:24:40.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.660+0000 7fa7ef254640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa7d8039020 0x7fa7d803b4e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:40.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.660+0000 7fa7ef254640 1 --2- 172.21.15.64:0/3571307335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7e8107510 0x7fa7e819b530 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:40.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.660+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 >> 172.21.15.64:0/3571307335 conn(0x7fa7e8102c60 msgr2=0x7fa7e8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:40.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.660+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 shutdown_connections 2024-08-19T01:24:40.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:39.660+0000 7fa7ef254640 1 -- 172.21.15.64:0/3571307335 wait complete. 2024-08-19T01:24:40.222 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T01:24:40.222 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-08-19T01:24:40.230 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T01:24:40.230 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-08-19T01:24:40.230 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-08-19T01:24:40.230 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-08-19T01:24:40.231 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph Dashboard is now available at: 2024-08-19T01:24:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:40.232 INFO:teuthology.orchestra.run.smithi064.stdout: URL: https://smithi064:8443/ 2024-08-19T01:24:40.232 INFO:teuthology.orchestra.run.smithi064.stdout: User: admin 2024-08-19T01:24:40.232 INFO:teuthology.orchestra.run.smithi064.stdout: Password: uxh1dpt6e3 2024-08-19T01:24:40.232 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:40.232 INFO:teuthology.orchestra.run.smithi064.stdout:Saving cluster configuration to /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config directory 2024-08-19T01:24:40.897 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:40 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3571307335' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.617+0000 7f33e60b7640 1 Processor -- start 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.617+0000 7f33e60b7640 1 -- start start 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.617+0000 7f33e60b7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e007c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.618+0000 7f33e60b7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f33e007c7e0 con 0x7f33e007ddf0 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.618+0000 7f33df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e007c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.618+0000 7f33df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e007c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43328/0 (socket says 172.21.15.64:43328) 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.618+0000 7f33df7fe640 1 -- 172.21.15.64:0/3104563065 learned_addr learned my addr 172.21.15.64:0/3104563065 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.618+0000 7f33df7fe640 1 -- 172.21.15.64:0/3104563065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e007c960 con 0x7f33e007ddf0 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.619+0000 7f33df7fe640 1 --2- 172.21.15.64:0/3104563065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e007c2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009820 tx=0x7f33d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73bde29fa017577 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.619+0000 7f33de7fc640 1 -- 172.21.15.64:0/3104563065 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d0037070 con 0x7f33e007ddf0 2024-08-19T01:24:41.453 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.619+0000 7f33de7fc640 1 -- 172.21.15.64:0/3104563065 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d0032920 con 0x7f33e007ddf0 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.619+0000 7f33de7fc640 1 -- 172.21.15.64:0/3104563065 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d0030810 con 0x7f33e007ddf0 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 -- 172.21.15.64:0/3104563065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 msgr2=0x7f33e007c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 --2- 172.21.15.64:0/3104563065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e007c2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009820 tx=0x7f33d002a270 comp rx=0 tx=0).stop 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 -- 172.21.15.64:0/3104563065 shutdown_connections 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 --2- 172.21.15.64:0/3104563065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e007c2a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 -- 172.21.15.64:0/3104563065 >> 172.21.15.64:0/3104563065 conn(0x7f33e0102c60 msgr2=0x7f33e01050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 -- 172.21.15.64:0/3104563065 shutdown_connections 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 -- 172.21.15.64:0/3104563065 wait complete. 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 Processor -- start 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.620+0000 7f33e60b7640 1 -- start start 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33e60b7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33e60b7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f33e010db00 con 0x7f33e007ddf0 2024-08-19T01:24:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e019d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43336/0 (socket says 172.21.15.64:43336) 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33df7fe640 1 -- 172.21.15.64:0/1535209504 learned_addr learned my addr 172.21.15.64:0/1535209504 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33df7fe640 1 -- 172.21.15.64:0/1535209504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e019dc20 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33df7fe640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e019d6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33d0030420 tx=0x7f33d0032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33dcff9640 1 -- 172.21.15.64:0/1535209504 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d0037040 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33dcff9640 1 -- 172.21.15.64:0/1535209504 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d0040070 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e019deb0 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33dcff9640 1 -- 172.21.15.64:0/1535209504 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d003a6e0 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.621+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e01a0ba0 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.622+0000 7f33dcff9640 1 -- 172.21.15.64:0/1535209504 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f33d003a9e0 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.622+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e007c320 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.622+0000 7f33dcff9640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f33cc039060 0x7f33cc03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.622+0000 7f33dcff9640 1 -- 172.21.15.64:0/1535209504 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f33d006c8f0 con 0x7f33e007ddf0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.623+0000 7f33deffd640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f33cc039060 0x7f33cc03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:41.455 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.623+0000 7f33deffd640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f33cc039060 0x7f33cc03b520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f33c80098c0 tx=0x7f33c8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.627+0000 7f33dcff9640 1 -- 172.21.15.64:0/1535209504 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f33d0039780 con 0x7f33e007ddf0 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.894+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f33e01a1190 con 0x7f33e007ddf0 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.896+0000 7f33dcff9640 1 -- 172.21.15.64:0/1535209504 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f33d003ae20 con 0x7f33e007ddf0 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f33cc039060 msgr2=0x7f33cc03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f33cc039060 0x7f33cc03b520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f33c80098c0 tx=0x7f33c8006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 msgr2=0x7f33e019d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e019d6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33d0030420 tx=0x7f33d0032670 comp rx=0 tx=0).stop 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 shutdown_connections 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f33cc039060 0x7f33cc03b520 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 --2- 172.21.15.64:0/1535209504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33e007ddf0 0x7f33e019d6e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 >> 172.21.15.64:0/1535209504 conn(0x7f33e0102c60 msgr2=0x7f33e0103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:41.456 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 shutdown_connections 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-08-19T01:24:40.898+0000 7f33e60b7640 1 -- 172.21.15.64:0/1535209504 wait complete. 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout:Or, if you are only running a single cluster on this host: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout:Please consider enabling telemetry to help improve Ceph: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: ceph telemetry on 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.457 INFO:teuthology.orchestra.run.smithi064.stdout:For more information see: 2024-08-19T01:24:41.458 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.458 INFO:teuthology.orchestra.run.smithi064.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-08-19T01:24:41.458 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:41.458 INFO:teuthology.orchestra.run.smithi064.stdout:Bootstrap complete. 2024-08-19T01:24:41.491 INFO:tasks.cephadm:Fetching config... 2024-08-19T01:24:41.491 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:24:41.491 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-08-19T01:24:41.507 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-08-19T01:24:41.507 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:24:41.507 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-08-19T01:24:41.564 INFO:tasks.cephadm:Fetching mon keyring... 2024-08-19T01:24:41.564 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:24:41.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/keyring of=/dev/stdout 2024-08-19T01:24:41.628 INFO:tasks.cephadm:Fetching pub ssh key... 2024-08-19T01:24:41.629 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:24:41.629 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-08-19T01:24:41.685 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-08-19T01:24:41.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoB69Z0HhChZZnW8q9333aR0h0iqtwWIXtkxv/ZlzkIPTWonq5gxRue5Qiqtl5pLM6KtcnDaFDdZOWG3wS7BRoP679a+23tNRTMycXRbjV9+veH+suPFZLM69zwP2TNE810Yb8LOAXxEqfxxn9+SEa2L5g9hBW9xwkg2yi6V8hrSFzAAx8rmdQfDKzpt3iqB3z4OzI0p+NIjqQAqALlsKabkYHN96KDtnUQ30AauFhISRoyx1o4M8yUdq7kdtABVOnK+4nD9wRXk33aghZ518zbEtOwSBhlExb41skQ8UHGd3Ms4RXoKtbq2olpxml4H/6pCR0WvO5A0Pk338AgYX6wHtN+b/+FuDE5nNsMR7fU3EN+fv9NmobfzFBAUzasLqvFJxf7LVDc68hjIr0H4aJGuYjfjoTaitOnSfo4imT/VvQLk8at6Zlq8QCtGlQD+2anRcXanFwChrRylTyI5zDTlO2wBrKxm/lO/KoBLsijKuFHUHGHZ8fO9L1Z4vg/s= ceph-88c1a048-5dc9-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-19T01:24:41.720 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:41 smithi064 ceph-mon[18515]: mgrmap e13: x(active, since 6s) 2024-08-19T01:24:41.720 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:41 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1535209504' entity='client.admin' 2024-08-19T01:24:41.757 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoB69Z0HhChZZnW8q9333aR0h0iqtwWIXtkxv/ZlzkIPTWonq5gxRue5Qiqtl5pLM6KtcnDaFDdZOWG3wS7BRoP679a+23tNRTMycXRbjV9+veH+suPFZLM69zwP2TNE810Yb8LOAXxEqfxxn9+SEa2L5g9hBW9xwkg2yi6V8hrSFzAAx8rmdQfDKzpt3iqB3z4OzI0p+NIjqQAqALlsKabkYHN96KDtnUQ30AauFhISRoyx1o4M8yUdq7kdtABVOnK+4nD9wRXk33aghZ518zbEtOwSBhlExb41skQ8UHGd3Ms4RXoKtbq2olpxml4H/6pCR0WvO5A0Pk338AgYX6wHtN+b/+FuDE5nNsMR7fU3EN+fv9NmobfzFBAUzasLqvFJxf7LVDc68hjIr0H4aJGuYjfjoTaitOnSfo4imT/VvQLk8at6Zlq8QCtGlQD+2anRcXanFwChrRylTyI5zDTlO2wBrKxm/lO/KoBLsijKuFHUHGHZ8fO9L1Z4vg/s= ceph-88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:24:41.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoB69Z0HhChZZnW8q9333aR0h0iqtwWIXtkxv/ZlzkIPTWonq5gxRue5Qiqtl5pLM6KtcnDaFDdZOWG3wS7BRoP679a+23tNRTMycXRbjV9+veH+suPFZLM69zwP2TNE810Yb8LOAXxEqfxxn9+SEa2L5g9hBW9xwkg2yi6V8hrSFzAAx8rmdQfDKzpt3iqB3z4OzI0p+NIjqQAqALlsKabkYHN96KDtnUQ30AauFhISRoyx1o4M8yUdq7kdtABVOnK+4nD9wRXk33aghZ518zbEtOwSBhlExb41skQ8UHGd3Ms4RXoKtbq2olpxml4H/6pCR0WvO5A0Pk338AgYX6wHtN+b/+FuDE5nNsMR7fU3EN+fv9NmobfzFBAUzasLqvFJxf7LVDc68hjIr0H4aJGuYjfjoTaitOnSfo4imT/VvQLk8at6Zlq8QCtGlQD+2anRcXanFwChrRylTyI5zDTlO2wBrKxm/lO/KoBLsijKuFHUHGHZ8fO9L1Z4vg/s= ceph-88c1a048-5dc9-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-19T01:24:41.811 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoB69Z0HhChZZnW8q9333aR0h0iqtwWIXtkxv/ZlzkIPTWonq5gxRue5Qiqtl5pLM6KtcnDaFDdZOWG3wS7BRoP679a+23tNRTMycXRbjV9+veH+suPFZLM69zwP2TNE810Yb8LOAXxEqfxxn9+SEa2L5g9hBW9xwkg2yi6V8hrSFzAAx8rmdQfDKzpt3iqB3z4OzI0p+NIjqQAqALlsKabkYHN96KDtnUQ30AauFhISRoyx1o4M8yUdq7kdtABVOnK+4nD9wRXk33aghZ518zbEtOwSBhlExb41skQ8UHGd3Ms4RXoKtbq2olpxml4H/6pCR0WvO5A0Pk338AgYX6wHtN+b/+FuDE5nNsMR7fU3EN+fv9NmobfzFBAUzasLqvFJxf7LVDc68hjIr0H4aJGuYjfjoTaitOnSfo4imT/VvQLk8at6Zlq8QCtGlQD+2anRcXanFwChrRylTyI5zDTlO2wBrKxm/lO/KoBLsijKuFHUHGHZ8fO9L1Z4vg/s= ceph-88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:24:41.823 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoB69Z0HhChZZnW8q9333aR0h0iqtwWIXtkxv/ZlzkIPTWonq5gxRue5Qiqtl5pLM6KtcnDaFDdZOWG3wS7BRoP679a+23tNRTMycXRbjV9+veH+suPFZLM69zwP2TNE810Yb8LOAXxEqfxxn9+SEa2L5g9hBW9xwkg2yi6V8hrSFzAAx8rmdQfDKzpt3iqB3z4OzI0p+NIjqQAqALlsKabkYHN96KDtnUQ30AauFhISRoyx1o4M8yUdq7kdtABVOnK+4nD9wRXk33aghZ518zbEtOwSBhlExb41skQ8UHGd3Ms4RXoKtbq2olpxml4H/6pCR0WvO5A0Pk338AgYX6wHtN+b/+FuDE5nNsMR7fU3EN+fv9NmobfzFBAUzasLqvFJxf7LVDc68hjIr0H4aJGuYjfjoTaitOnSfo4imT/VvQLk8at6Zlq8QCtGlQD+2anRcXanFwChrRylTyI5zDTlO2wBrKxm/lO/KoBLsijKuFHUHGHZ8fO9L1Z4vg/s= ceph-88c1a048-5dc9-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-19T01:24:41.866 INFO:teuthology.orchestra.run.smithi161.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoB69Z0HhChZZnW8q9333aR0h0iqtwWIXtkxv/ZlzkIPTWonq5gxRue5Qiqtl5pLM6KtcnDaFDdZOWG3wS7BRoP679a+23tNRTMycXRbjV9+veH+suPFZLM69zwP2TNE810Yb8LOAXxEqfxxn9+SEa2L5g9hBW9xwkg2yi6V8hrSFzAAx8rmdQfDKzpt3iqB3z4OzI0p+NIjqQAqALlsKabkYHN96KDtnUQ30AauFhISRoyx1o4M8yUdq7kdtABVOnK+4nD9wRXk33aghZ518zbEtOwSBhlExb41skQ8UHGd3Ms4RXoKtbq2olpxml4H/6pCR0WvO5A0Pk338AgYX6wHtN+b/+FuDE5nNsMR7fU3EN+fv9NmobfzFBAUzasLqvFJxf7LVDc68hjIr0H4aJGuYjfjoTaitOnSfo4imT/VvQLk8at6Zlq8QCtGlQD+2anRcXanFwChrRylTyI5zDTlO2wBrKxm/lO/KoBLsijKuFHUHGHZ8fO9L1Z4vg/s= ceph-88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:24:41.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-08-19T01:24:42.135 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:24:42.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.620+0000 7f645a964640 1 -- 172.21.15.64:0/1947148359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 msgr2=0x7f6454101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:42.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.620+0000 7f645a964640 1 --2- 172.21.15.64:0/1947148359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454101710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6444009880 tx=0x7f644402a5d0 comp rx=0 tx=0).stop 2024-08-19T01:24:42.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.620+0000 7f645a964640 1 -- 172.21.15.64:0/1947148359 shutdown_connections 2024-08-19T01:24:42.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.620+0000 7f645a964640 1 --2- 172.21.15.64:0/1947148359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454101710 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:42.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.620+0000 7f645a964640 1 -- 172.21.15.64:0/1947148359 >> 172.21.15.64:0/1947148359 conn(0x7f64540fcaa0 msgr2=0x7f64540feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.620+0000 7f645a964640 1 -- 172.21.15.64:0/1947148359 shutdown_connections 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.620+0000 7f645a964640 1 -- 172.21.15.64:0/1947148359 wait complete. 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.621+0000 7f645a964640 1 Processor -- start 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.621+0000 7f645a964640 1 -- start start 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.621+0000 7f645a964640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.621+0000 7f645a964640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6454107420 con 0x7f6454101310 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.621+0000 7f6453fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.621+0000 7f6453fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43350/0 (socket says 172.21.15.64:43350) 2024-08-19T01:24:42.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.621+0000 7f6453fff640 1 -- 172.21.15.64:0/3500598701 learned_addr learned my addr 172.21.15.64:0/3500598701 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:42.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.622+0000 7f6453fff640 1 -- 172.21.15.64:0/3500598701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6454197000 con 0x7f6454101310 2024-08-19T01:24:42.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.622+0000 7f6453fff640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454196ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6444004f60 tx=0x7f64440048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:42.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.622+0000 7f64517fa640 1 -- 172.21.15.64:0/3500598701 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f644403f070 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.622+0000 7f64517fa640 1 -- 172.21.15.64:0/3500598701 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6444004d40 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.622+0000 7f64517fa640 1 -- 172.21.15.64:0/3500598701 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6444035030 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.622+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6454197290 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.622+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6454199f80 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.623+0000 7f64517fa640 1 -- 172.21.15.64:0/3500598701 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f6444042070 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.623+0000 7f64517fa640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f643c039060 0x7f643c03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.623+0000 7f64517fa640 1 -- 172.21.15.64:0/3500598701 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f644406cd40 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.623+0000 7f64537fe640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f643c039060 0x7f643c03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.623+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6454107ba0 con 0x7f6454101310 2024-08-19T01:24:42.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.624+0000 7f64537fe640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f643c039060 0x7f643c03b520 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f64480098c0 tx=0x7f6448006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:42.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.628+0000 7f64517fa640 1 -- 172.21.15.64:0/3500598701 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6444042360 con 0x7f6454101310 2024-08-19T01:24:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.808+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f645418fc50 con 0x7f6454101310 2024-08-19T01:24:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.811+0000 7f64517fa640 1 -- 172.21.15.64:0/3500598701 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f644402f300 con 0x7f6454101310 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f643c039060 msgr2=0x7f643c03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f643c039060 0x7f643c03b520 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f64480098c0 tx=0x7f6448006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 msgr2=0x7f6454196ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454196ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6444004f60 tx=0x7f64440048f0 comp rx=0 tx=0).stop 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 shutdown_connections 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f643c039060 0x7f643c03b520 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 --2- 172.21.15.64:0/3500598701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6454101310 0x7f6454196ac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 >> 172.21.15.64:0/3500598701 conn(0x7f64540fcaa0 msgr2=0x7f64540fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 shutdown_connections 2024-08-19T01:24:42.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:42.814+0000 7f645a964640 1 -- 172.21.15.64:0/3500598701 wait complete. 2024-08-19T01:24:43.796 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-08-19T01:24:43.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-08-19T01:24:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:43 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3500598701' entity='client.admin' 2024-08-19T01:24:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:24:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:44.047 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:24:44.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 -- 172.21.15.64:0/4058958450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 msgr2=0x7f67b8100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:44.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 --2- 172.21.15.64:0/4058958450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b8100670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009880 tx=0x7f67b402a5d0 comp rx=0 tx=0).stop 2024-08-19T01:24:44.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 -- 172.21.15.64:0/4058958450 shutdown_connections 2024-08-19T01:24:44.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 --2- 172.21.15.64:0/4058958450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b8100670 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:44.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 -- 172.21.15.64:0/4058958450 >> 172.21.15.64:0/4058958450 conn(0x7f67b80fba00 msgr2=0x7f67b80fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:44.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 -- 172.21.15.64:0/4058958450 shutdown_connections 2024-08-19T01:24:44.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 -- 172.21.15.64:0/4058958450 wait complete. 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.473+0000 7f67c066c640 1 Processor -- start 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67c066c640 1 -- start start 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67c066c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b80fd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67c066c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f67b8106320 con 0x7f67b8100270 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67be408640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b80fd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67be408640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b80fd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43374/0 (socket says 172.21.15.64:43374) 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67be408640 1 -- 172.21.15.64:0/2408803853 learned_addr learned my addr 172.21.15.64:0/2408803853 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67be408640 1 -- 172.21.15.64:0/2408803853 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b80fde00 con 0x7f67b8100270 2024-08-19T01:24:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67be408640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b80fd8c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67b4004f60 tx=0x7f67b40048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:44.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67af7fe640 1 -- 172.21.15.64:0/2408803853 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67b403f070 con 0x7f67b8100270 2024-08-19T01:24:44.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67af7fe640 1 -- 172.21.15.64:0/2408803853 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b4004d40 con 0x7f67b8100270 2024-08-19T01:24:44.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.474+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b80fbff0 con 0x7f67b8100270 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.475+0000 7f67af7fe640 1 -- 172.21.15.64:0/2408803853 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67b4035030 con 0x7f67b8100270 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.475+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b80fc530 con 0x7f67b8100270 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.475+0000 7f67af7fe640 1 -- 172.21.15.64:0/2408803853 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f67b4042070 con 0x7f67b8100270 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.476+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b8106aa0 con 0x7f67b8100270 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.476+0000 7f67af7fe640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f67a40390c0 0x7f67a403b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.476+0000 7f67af7fe640 1 -- 172.21.15.64:0/2408803853 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f67b406cca0 con 0x7f67b8100270 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.476+0000 7f67bdc07640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f67a40390c0 0x7f67a403b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:44.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.476+0000 7f67bdc07640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f67a40390c0 0x7f67a403b580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67a80098c0 tx=0x7f67a8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:44.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.480+0000 7f67af7fe640 1 -- 172.21.15.64:0/2408803853 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f67b404a260 con 0x7f67b8100270 2024-08-19T01:24:44.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.658+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f67b818ffe0 con 0x7f67a40390c0 2024-08-19T01:24:44.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.662+0000 7f67af7fe640 1 -- 172.21.15.64:0/2408803853 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f67b818ffe0 con 0x7f67a40390c0 2024-08-19T01:24:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f67a40390c0 msgr2=0x7f67a403b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f67a40390c0 0x7f67a403b580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67a80098c0 tx=0x7f67a8006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 msgr2=0x7f67b80fd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b80fd8c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67b4004f60 tx=0x7f67b40048f0 comp rx=0 tx=0).stop 2024-08-19T01:24:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 shutdown_connections 2024-08-19T01:24:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f67a40390c0 0x7f67a403b580 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:44.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 --2- 172.21.15.64:0/2408803853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67b8100270 0x7f67b80fd8c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:44.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 >> 172.21.15.64:0/2408803853 conn(0x7f67b80fba00 msgr2=0x7f67b80698a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:44.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 shutdown_connections 2024-08-19T01:24:44.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:44.664+0000 7f67c066c640 1 -- 172.21.15.64:0/2408803853 wait complete. 2024-08-19T01:24:45.410 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi123 2024-08-19T01:24:45.410 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:24:45.410 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.conf 2024-08-19T01:24:45.429 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:24:45.429 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:24:45.486 INFO:tasks.cephadm:Adding host smithi123 to orchestrator... 2024-08-19T01:24:45.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch host add smithi123 2024-08-19T01:24:45.741 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:24:45.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:45.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:45.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:45.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:24:45.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:24:45.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: Updating smithi064:/etc/ceph/ceph.conf 2024-08-19T01:24:45.958 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:24:45.959 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:24:45.959 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:24:45.959 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:45.959 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:45.959 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:46.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.153+0000 7fdce199c640 1 -- 172.21.15.64:0/3347039477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 msgr2=0x7fdcdc100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:46.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 --2- 172.21.15.64:0/3347039477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc100800 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009880 tx=0x7fdccc02a5d0 comp rx=0 tx=0).stop 2024-08-19T01:24:46.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 -- 172.21.15.64:0/3347039477 shutdown_connections 2024-08-19T01:24:46.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 --2- 172.21.15.64:0/3347039477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc100800 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:46.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 -- 172.21.15.64:0/3347039477 >> 172.21.15.64:0/3347039477 conn(0x7fdcdc0fbee0 msgr2=0x7fdcdc0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 -- 172.21.15.64:0/3347039477 shutdown_connections 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 -- 172.21.15.64:0/3347039477 wait complete. 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 Processor -- start 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 -- start start 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdce199c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc10a870 con 0x7fdcdc100420 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.154+0000 7fdcdaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdcdaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43392/0 (socket says 172.21.15.64:43392) 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdcdaffd640 1 -- 172.21.15.64:0/2472321776 learned_addr learned my addr 172.21.15.64:0/2472321776 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdcdaffd640 1 -- 172.21.15.64:0/2472321776 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc1973f0 con 0x7fdcdc100420 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdcdaffd640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc196eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009230 tx=0x7fdccc003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:46.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdce099a640 1 -- 172.21.15.64:0/2472321776 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdccc03f070 con 0x7fdcdc100420 2024-08-19T01:24:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdce099a640 1 -- 172.21.15.64:0/2472321776 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc03bc80 con 0x7fdcdc100420 2024-08-19T01:24:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcdc1976e0 con 0x7fdcdc100420 2024-08-19T01:24:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdce099a640 1 -- 172.21.15.64:0/2472321776 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdccc035030 con 0x7fdcdc100420 2024-08-19T01:24:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.155+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcdc19b780 con 0x7fdcdc100420 2024-08-19T01:24:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.156+0000 7fdce099a640 1 -- 172.21.15.64:0/2472321776 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fdccc042070 con 0x7fdcdc100420 2024-08-19T01:24:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.156+0000 7fdce099a640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc80390c0 0x7fdcc803b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.156+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcdc101900 con 0x7fdcdc100420 2024-08-19T01:24:46.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.156+0000 7fdce099a640 1 -- 172.21.15.64:0/2472321776 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdccc06cd10 con 0x7fdcdc100420 2024-08-19T01:24:46.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.156+0000 7fdcd3fff640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc80390c0 0x7fdcc803b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:46.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.157+0000 7fdcd3fff640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc80390c0 0x7fdcc803b580 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098c0 tx=0x7fdcc4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:46.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.161+0000 7fdce099a640 1 -- 172.21.15.64:0/2472321776 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fdccc04c350 con 0x7fdcdc100420 2024-08-19T01:24:46.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:46.340+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}) -- 0x7fdcdc198210 con 0x7fdcc80390c0 2024-08-19T01:24:46.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:46 smithi064 ceph-mon[18515]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:47.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:47 smithi064 ceph-mon[18515]: Deploying cephadm binary to smithi123 2024-08-19T01:24:50.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.173+0000 7fdce099a640 1 -- 172.21.15.64:0/2472321776 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fdcdc198210 con 0x7fdcc80390c0 2024-08-19T01:24:50.176 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi123' with addr '172.21.15.123' 2024-08-19T01:24:50.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc80390c0 msgr2=0x7fdcc803b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:50.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc80390c0 0x7fdcc803b580 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098c0 tx=0x7fdcc4006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:50.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 msgr2=0x7fdcdc196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:50.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc196eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009230 tx=0x7fdccc003d40 comp rx=0 tx=0).stop 2024-08-19T01:24:50.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 shutdown_connections 2024-08-19T01:24:50.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc80390c0 0x7fdcc803b580 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:50.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 --2- 172.21.15.64:0/2472321776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcdc100420 0x7fdcdc196eb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:50.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 >> 172.21.15.64:0/2472321776 conn(0x7fdcdc0fbee0 msgr2=0x7fdcdc103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:50.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 shutdown_connections 2024-08-19T01:24:50.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:50.176+0000 7fdce199c640 1 -- 172.21.15.64:0/2472321776 wait complete. 2024-08-19T01:24:50.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch host ls --format=json 2024-08-19T01:24:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:24:51.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:51.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:51 smithi064 ceph-mon[18515]: Added host smithi123 2024-08-19T01:24:51.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:51.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:51.465 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:51.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.468+0000 7f42cf98a640 1 -- 172.21.15.64:0/558553199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 msgr2=0x7f42c8100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:51.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.468+0000 7f42cf98a640 1 --2- 172.21.15.64:0/558553199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8100800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009880 tx=0x7f42c402a5d0 comp rx=0 tx=0).stop 2024-08-19T01:24:51.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.468+0000 7f42cf98a640 1 -- 172.21.15.64:0/558553199 shutdown_connections 2024-08-19T01:24:51.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.468+0000 7f42cf98a640 1 --2- 172.21.15.64:0/558553199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8100800 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:51.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.468+0000 7f42cf98a640 1 -- 172.21.15.64:0/558553199 >> 172.21.15.64:0/558553199 conn(0x7f42c80fbee0 msgr2=0x7f42c80fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:51.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.468+0000 7f42cf98a640 1 -- 172.21.15.64:0/558553199 shutdown_connections 2024-08-19T01:24:51.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.468+0000 7f42cf98a640 1 -- 172.21.15.64:0/558553199 wait complete. 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cf98a640 1 Processor -- start 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cf98a640 1 -- start start 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cf98a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cf98a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f42c810a870 con 0x7f42c8100420 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cd726640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cd726640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39066/0 (socket says 172.21.15.64:39066) 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cd726640 1 -- 172.21.15.64:0/2394709305 learned_addr learned my addr 172.21.15.64:0/2394709305 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.469+0000 7f42cd726640 1 -- 172.21.15.64:0/2394709305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c81973f0 con 0x7f42c8100420 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.470+0000 7f42cd726640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8196eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009230 tx=0x7f42c4003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.470+0000 7f42be7fc640 1 -- 172.21.15.64:0/2394709305 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c403f070 con 0x7f42c8100420 2024-08-19T01:24:51.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.470+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c81976e0 con 0x7f42c8100420 2024-08-19T01:24:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.470+0000 7f42be7fc640 1 -- 172.21.15.64:0/2394709305 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42c403bc80 con 0x7f42c8100420 2024-08-19T01:24:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.470+0000 7f42be7fc640 1 -- 172.21.15.64:0/2394709305 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c4035030 con 0x7f42c8100420 2024-08-19T01:24:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.470+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c819b780 con 0x7f42c8100420 2024-08-19T01:24:51.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.471+0000 7f42be7fc640 1 -- 172.21.15.64:0/2394709305 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f42c4042070 con 0x7f42c8100420 2024-08-19T01:24:51.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.471+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c8101900 con 0x7f42c8100420 2024-08-19T01:24:51.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.471+0000 7f42be7fc640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42b4039060 0x7f42b403b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:51.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.471+0000 7f42be7fc640 1 -- 172.21.15.64:0/2394709305 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f42c406cd80 con 0x7f42c8100420 2024-08-19T01:24:51.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.471+0000 7f42ccf25640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42b4039060 0x7f42b403b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:51.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.471+0000 7f42ccf25640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42b4039060 0x7f42b403b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f42b80098c0 tx=0x7f42b8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:51.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.476+0000 7f42be7fc640 1 -- 172.21.15.64:0/2394709305 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f42c4031570 con 0x7f42c8100420 2024-08-19T01:24:51.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.657+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f42c8100880 con 0x7f42b4039060 2024-08-19T01:24:51.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.658+0000 7f42be7fc640 1 -- 172.21.15.64:0/2394709305 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f42c8100880 con 0x7f42b4039060 2024-08-19T01:24:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:51.660 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}] 2024-08-19T01:24:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42b4039060 msgr2=0x7f42b403b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42b4039060 0x7f42b403b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f42b80098c0 tx=0x7f42b8006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 msgr2=0x7f42c8196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8196eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009230 tx=0x7f42c4003d40 comp rx=0 tx=0).stop 2024-08-19T01:24:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 shutdown_connections 2024-08-19T01:24:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42b4039060 0x7f42b403b520 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 --2- 172.21.15.64:0/2394709305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42c8100420 0x7f42c8196eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:51.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 >> 172.21.15.64:0/2394709305 conn(0x7f42c80fbee0 msgr2=0x7f42c8103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:51.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 shutdown_connections 2024-08-19T01:24:51.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:51.660+0000 7f42cf98a640 1 -- 172.21.15.64:0/2394709305 wait complete. 2024-08-19T01:24:52.229 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi161 2024-08-19T01:24:52.229 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:24:52.229 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.conf 2024-08-19T01:24:52.249 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:24:52.249 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:24:52.306 INFO:tasks.cephadm:Adding host smithi161 to orchestrator... 2024-08-19T01:24:52.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch host add smithi161 2024-08-19T01:24:52.556 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:24:52.778 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:52 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.942+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/3629938862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 msgr2=0x7fd7ac101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.942+0000 7fd7b2ae4640 1 --2- 172.21.15.64:0/3629938862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac101900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009c20 tx=0x7fd79c02aa90 comp rx=0 tx=0).stop 2024-08-19T01:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.942+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/3629938862 shutdown_connections 2024-08-19T01:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.942+0000 7fd7b2ae4640 1 --2- 172.21.15.64:0/3629938862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac101900 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.942+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/3629938862 >> 172.21.15.64:0/3629938862 conn(0x7fd7ac0fcfc0 msgr2=0x7fd7ac0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.942+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/3629938862 shutdown_connections 2024-08-19T01:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.942+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/3629938862 wait complete. 2024-08-19T01:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b2ae4640 1 Processor -- start 2024-08-19T01:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b2ae4640 1 -- start start 2024-08-19T01:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b2ae4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b2ae4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac10b970 con 0x7fd7ac101520 2024-08-19T01:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b0880640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b0880640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39080/0 (socket says 172.21.15.64:39080) 2024-08-19T01:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b0880640 1 -- 172.21.15.64:0/649653302 learned_addr learned my addr 172.21.15.64:0/649653302 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b0880640 1 -- 172.21.15.64:0/649653302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac197410 con 0x7fd7ac101520 2024-08-19T01:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.943+0000 7fd7b0880640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac196ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005e30 tx=0x7fd79c02fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.944+0000 7fd7a9ffb640 1 -- 172.21.15.64:0/649653302 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd79c047070 con 0x7fd7ac101520 2024-08-19T01:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.944+0000 7fd7a9ffb640 1 -- 172.21.15.64:0/649653302 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79c003df0 con 0x7fd7ac101520 2024-08-19T01:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.944+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac197700 con 0x7fd7ac101520 2024-08-19T01:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.944+0000 7fd7a9ffb640 1 -- 172.21.15.64:0/649653302 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd79c033050 con 0x7fd7ac101520 2024-08-19T01:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.944+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac19b7a0 con 0x7fd7ac101520 2024-08-19T01:24:52.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.945+0000 7fd7a9ffb640 1 -- 172.21.15.64:0/649653302 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd79c04a070 con 0x7fd7ac101520 2024-08-19T01:24:52.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.945+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ac102a00 con 0x7fd7ac101520 2024-08-19T01:24:52.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.945+0000 7fd7a9ffb640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7980390c0 0x7fd79803b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:52.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.945+0000 7fd7a9ffb640 1 -- 172.21.15.64:0/649653302 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd79c03e030 con 0x7fd7ac101520 2024-08-19T01:24:52.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.945+0000 7fd7abfff640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7980390c0 0x7fd79803b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:52.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.945+0000 7fd7abfff640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7980390c0 0x7fd79803b580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd7940098c0 tx=0x7fd794006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:52.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:52.950+0000 7fd7a9ffb640 1 -- 172.21.15.64:0/649653302 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd79c04a360 con 0x7fd7ac101520 2024-08-19T01:24:53.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:53.130+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}) -- 0x7fd7ac1019e0 con 0x7fd7980390c0 2024-08-19T01:24:53.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:53 smithi064 ceph-mon[18515]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:24:54.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:24:54.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:54.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:54.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:54.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:54.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:24:54.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:24:54.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:55 smithi064 ceph-mon[18515]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:55 smithi064 ceph-mon[18515]: Updating smithi123:/etc/ceph/ceph.conf 2024-08-19T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:55 smithi064 ceph-mon[18515]: Deploying cephadm binary to smithi161 2024-08-19T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:55 smithi064 ceph-mon[18515]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:56.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:56 smithi064 ceph-mon[18515]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:24:56.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:56 smithi064 ceph-mon[18515]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:24:57.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.144+0000 7fd7a9ffb640 1 -- 172.21.15.64:0/649653302 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd7ac1019e0 con 0x7fd7980390c0 2024-08-19T01:24:57.147 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi161' with addr '172.21.15.161' 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.147+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7980390c0 msgr2=0x7fd79803b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.147+0000 7fd7b2ae4640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7980390c0 0x7fd79803b580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd7940098c0 tx=0x7fd794006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 msgr2=0x7fd7ac196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac196ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005e30 tx=0x7fd79c02fc10 comp rx=0 tx=0).stop 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 shutdown_connections 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7980390c0 0x7fd79803b580 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 --2- 172.21.15.64:0/649653302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac101520 0x7fd7ac196ed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 >> 172.21.15.64:0/649653302 conn(0x7fd7ac0fcfc0 msgr2=0x7fd7ac108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 shutdown_connections 2024-08-19T01:24:57.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:57.148+0000 7fd7b2ae4640 1 -- 172.21.15.64:0/649653302 wait complete. 2024-08-19T01:24:57.824 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:57 smithi064 ceph-mon[18515]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:24:57.824 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:57.824 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:24:57.824 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:57.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch host ls --format=json 2024-08-19T01:24:58.098 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:24:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.481+0000 7f28e4cdb640 1 -- 172.21.15.64:0/373549230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 msgr2=0x7f28e0100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.481+0000 7f28e4cdb640 1 --2- 172.21.15.64:0/373549230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0100800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009c20 tx=0x7f28d802aa90 comp rx=0 tx=0).stop 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.481+0000 7f28e4cdb640 1 -- 172.21.15.64:0/373549230 shutdown_connections 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.481+0000 7f28e4cdb640 1 --2- 172.21.15.64:0/373549230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0100800 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.481+0000 7f28e4cdb640 1 -- 172.21.15.64:0/373549230 >> 172.21.15.64:0/373549230 conn(0x7f28e00fbec0 msgr2=0x7f28e00fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.481+0000 7f28e4cdb640 1 -- 172.21.15.64:0/373549230 shutdown_connections 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.481+0000 7f28e4cdb640 1 -- 172.21.15.64:0/373549230 wait complete. 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.482+0000 7f28e4cdb640 1 Processor -- start 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.482+0000 7f28e4cdb640 1 -- start start 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.482+0000 7f28e4cdb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.482+0000 7f28e4cdb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010a870 con 0x7f28e0100420 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.482+0000 7f28de59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.482+0000 7f28de59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0197cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39096/0 (socket says 172.21.15.64:39096) 2024-08-19T01:24:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28de59c640 1 -- 172.21.15.64:0/3804859503 learned_addr learned my addr 172.21.15.64:0/3804859503 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28de59c640 1 -- 172.21.15.64:0/3804859503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e0198210 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28de59c640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0197cd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28d8005e30 tx=0x7f28d802fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28d77fe640 1 -- 172.21.15.64:0/3804859503 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8047070 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28d77fe640 1 -- 172.21.15.64:0/3804859503 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d8003df0 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28d77fe640 1 -- 172.21.15.64:0/3804859503 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8033050 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e0198500 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.483+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e01989c0 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.484+0000 7f28d77fe640 1 -- 172.21.15.64:0/3804859503 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f28d804a070 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.484+0000 7f28d77fe640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28cc0390c0 0x7f28cc03b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.484+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e0101900 con 0x7f28e0100420 2024-08-19T01:24:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.484+0000 7f28d77fe640 1 -- 172.21.15.64:0/3804859503 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f28d803e030 con 0x7f28e0100420 2024-08-19T01:24:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.484+0000 7f28ddd9b640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28cc0390c0 0x7f28cc03b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.485+0000 7f28ddd9b640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28cc0390c0 0x7f28cc03b580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f28c80098c0 tx=0x7f28c8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:58.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.489+0000 7f28d77fe640 1 -- 172.21.15.64:0/3804859503 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f28d80373b0 con 0x7f28e0100420 2024-08-19T01:24:58.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.669+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f28e0100880 con 0x7f28cc0390c0 2024-08-19T01:24:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.670+0000 7f28d77fe640 1 -- 172.21.15.64:0/3804859503 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f28e0100880 con 0x7f28cc0390c0 2024-08-19T01:24:58.672 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:24:58.673 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "172.21.15.161", "hostname": "smithi161", "labels": [], "status": ""}] 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28cc0390c0 msgr2=0x7f28cc03b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28cc0390c0 0x7f28cc03b580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f28c80098c0 tx=0x7f28c8006db0 comp rx=0 tx=0).stop 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 msgr2=0x7f28e0197cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0197cd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28d8005e30 tx=0x7f28d802fc10 comp rx=0 tx=0).stop 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 shutdown_connections 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28cc0390c0 0x7f28cc03b580 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 --2- 172.21.15.64:0/3804859503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28e0100420 0x7f28e0197cd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 >> 172.21.15.64:0/3804859503 conn(0x7f28e00fbec0 msgr2=0x7f28e0107e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 shutdown_connections 2024-08-19T01:24:58.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:58.673+0000 7f28e4cdb640 1 -- 172.21.15.64:0/3804859503 wait complete. 2024-08-19T01:24:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:58 smithi064 ceph-mon[18515]: Added host smithi161 2024-08-19T01:24:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:58.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:24:59.294 INFO:tasks.cephadm:Setting crush tunables to default 2024-08-19T01:24:59.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd crush tunables default 2024-08-19T01:24:59.545 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:24:59.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:24:59 smithi064 ceph-mon[18515]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:24:59.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.956+0000 7f124e75e640 1 -- 172.21.15.64:0/2852529712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 msgr2=0x7f12480ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:24:59.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.956+0000 7f124e75e640 1 --2- 172.21.15.64:0/2852529712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f12480ffe40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1238009880 tx=0x7f123802a5d0 comp rx=0 tx=0).stop 2024-08-19T01:24:59.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.956+0000 7f124e75e640 1 -- 172.21.15.64:0/2852529712 shutdown_connections 2024-08-19T01:24:59.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.956+0000 7f124e75e640 1 --2- 172.21.15.64:0/2852529712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f12480ffe40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:24:59.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.956+0000 7f124e75e640 1 -- 172.21.15.64:0/2852529712 >> 172.21.15.64:0/2852529712 conn(0x7f12480fb650 msgr2=0x7f12480fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:24:59.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.956+0000 7f124e75e640 1 -- 172.21.15.64:0/2852529712 shutdown_connections 2024-08-19T01:24:59.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.956+0000 7f124e75e640 1 -- 172.21.15.64:0/2852529712 wait complete. 2024-08-19T01:24:59.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.957+0000 7f124e75e640 1 Processor -- start 2024-08-19T01:24:59.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.957+0000 7f124e75e640 1 -- start start 2024-08-19T01:24:59.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.957+0000 7f124e75e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f1248194c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:59.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.957+0000 7f124e75e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f124810b980 con 0x7f12480ffa60 2024-08-19T01:24:59.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.957+0000 7f1247fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f1248194c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:59.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.957+0000 7f1247fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f1248194c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36394/0 (socket says 172.21.15.64:36394) 2024-08-19T01:24:59.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.957+0000 7f1247fff640 1 -- 172.21.15.64:0/3681431991 learned_addr learned my addr 172.21.15.64:0/3681431991 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:24:59.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.958+0000 7f1247fff640 1 -- 172.21.15.64:0/3681431991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1248195170 con 0x7f12480ffa60 2024-08-19T01:24:59.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.958+0000 7f1247fff640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f1248194c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1238009230 tx=0x7f1238003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:59.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.958+0000 7f12457fa640 1 -- 172.21.15.64:0/3681431991 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f123803f070 con 0x7f12480ffa60 2024-08-19T01:24:59.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.958+0000 7f12457fa640 1 -- 172.21.15.64:0/3681431991 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f123803bc80 con 0x7f12480ffa60 2024-08-19T01:24:59.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.958+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1248195460 con 0x7f12480ffa60 2024-08-19T01:24:59.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.958+0000 7f12457fa640 1 -- 172.21.15.64:0/3681431991 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1238035030 con 0x7f12480ffa60 2024-08-19T01:24:59.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.958+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12481994f0 con 0x7f12480ffa60 2024-08-19T01:24:59.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.959+0000 7f12457fa640 1 -- 172.21.15.64:0/3681431991 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1238042070 con 0x7f12480ffa60 2024-08-19T01:24:59.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.959+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1248102b20 con 0x7f12480ffa60 2024-08-19T01:24:59.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.959+0000 7f12457fa640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1230039060 0x7f123003b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:24:59.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.959+0000 7f12457fa640 1 -- 172.21.15.64:0/3681431991 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f123806cd10 con 0x7f12480ffa60 2024-08-19T01:24:59.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.959+0000 7f12477fe640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1230039060 0x7f123003b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:24:59.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.960+0000 7f12477fe640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1230039060 0x7f123003b520 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f123c0098c0 tx=0x7f123c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:24:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:24:59.964+0000 7f12457fa640 1 -- 172.21.15.64:0/3681431991 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1238072020 con 0x7f12480ffa60 2024-08-19T01:25:00.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.144+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f12480ffec0 con 0x7f12480ffa60 2024-08-19T01:25:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.532+0000 7f12457fa640 1 -- 172.21.15.64:0/3681431991 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f123804c350 con 0x7f12480ffa60 2024-08-19T01:25:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-08-19T01:25:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1230039060 msgr2=0x7f123003b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1230039060 0x7f123003b520 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f123c0098c0 tx=0x7f123c006db0 comp rx=0 tx=0).stop 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 msgr2=0x7f1248194c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f1248194c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1238009230 tx=0x7f1238003d40 comp rx=0 tx=0).stop 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 shutdown_connections 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1230039060 0x7f123003b520 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 --2- 172.21.15.64:0/3681431991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12480ffa60 0x7f1248194c30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.534+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 >> 172.21.15.64:0/3681431991 conn(0x7f12480fb650 msgr2=0x7f1248109400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.535+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 shutdown_connections 2024-08-19T01:25:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:00.535+0000 7f124e75e640 1 -- 172.21.15.64:0/3681431991 wait complete. 2024-08-19T01:25:00.833 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:00 smithi064 ceph-mon[18515]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:25:00.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:00 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3681431991' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-19T01:25:01.108 INFO:tasks.cephadm:Adding mon.a on smithi064 2024-08-19T01:25:01.108 INFO:tasks.cephadm:Adding mon.b on smithi123 2024-08-19T01:25:01.108 INFO:tasks.cephadm:Adding mon.c on smithi161 2024-08-19T01:25:01.108 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch apply mon '3;smithi064:172.21.15.64=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c' 2024-08-19T01:25:01.358 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-19T01:25:01.406 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3681431991' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:01.858 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:01.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.882+0000 7f1adb0db640 1 -- 172.21.15.161:0/2284709044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 msgr2=0x7f1ad4104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:01.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.882+0000 7f1adb0db640 1 --2- 172.21.15.161:0/2284709044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4104250 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009880 tx=0x7f1ac802a5d0 comp rx=0 tx=0).stop 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.882+0000 7f1adb0db640 1 -- 172.21.15.161:0/2284709044 shutdown_connections 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.882+0000 7f1adb0db640 1 --2- 172.21.15.161:0/2284709044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4104250 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.882+0000 7f1adb0db640 1 -- 172.21.15.161:0/2284709044 >> 172.21.15.161:0/2284709044 conn(0x7f1ad40fba00 msgr2=0x7f1ad40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.882+0000 7f1adb0db640 1 -- 172.21.15.161:0/2284709044 shutdown_connections 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.882+0000 7f1adb0db640 1 -- 172.21.15.161:0/2284709044 wait complete. 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1adb0db640 1 Processor -- start 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1adb0db640 1 -- start start 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1adb0db640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1adb0db640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad4107410 con 0x7f1ad4101e60 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1ad8e77640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1ad8e77640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.161:45740/0 (socket says 172.21.15.161:45740) 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1ad8e77640 1 -- 172.21.15.161:0/100309777 learned_addr learned my addr 172.21.15.161:0/100309777 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:25:01.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1ad8e77640 1 -- 172.21.15.161:0/100309777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad4196ff0 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.883+0000 7f1ad8e77640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4196ab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8004f60 tx=0x7f1ac80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.884+0000 7f1ad1ffb640 1 -- 172.21.15.161:0/100309777 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ac803f070 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.884+0000 7f1ad1ffb640 1 -- 172.21.15.161:0/100309777 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac8003ad0 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.884+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad4197280 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.884+0000 7f1ad1ffb640 1 -- 172.21.15.161:0/100309777 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ac8035030 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.884+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad4199f70 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.885+0000 7f1ad1ffb640 1 -- 172.21.15.161:0/100309777 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1ac8042070 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.885+0000 7f1ad1ffb640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1ac4039060 0x7f1ac403b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.885+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad4107b90 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.885+0000 7f1ad1ffb640 1 -- 172.21.15.161:0/100309777 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1ac806d130 con 0x7f1ad4101e60 2024-08-19T01:25:01.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.885+0000 7f1ad3fff640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1ac4039060 0x7f1ac403b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:01.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.886+0000 7f1ad3fff640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1ac4039060 0x7f1ac403b520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00098c0 tx=0x7f1ac0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:01.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:01.890+0000 7f1ad1ffb640 1 -- 172.21.15.161:0/100309777 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1ac802d1c0 con 0x7f1ad4101e60 2024-08-19T01:25:02.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.080+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}) -- 0x7f1ad402d710 con 0x7f1ac4039060 2024-08-19T01:25:02.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.085+0000 7f1ad1ffb640 1 -- 172.21.15.161:0/100309777 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1ad402d710 con 0x7f1ac4039060 2024-08-19T01:25:02.087 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mon update... 2024-08-19T01:25:02.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1ac4039060 msgr2=0x7f1ac403b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:02.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1ac4039060 0x7f1ac403b520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00098c0 tx=0x7f1ac0006db0 comp rx=0 tx=0).stop 2024-08-19T01:25:02.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 msgr2=0x7f1ad4196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:02.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4196ab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8004f60 tx=0x7f1ac80048f0 comp rx=0 tx=0).stop 2024-08-19T01:25:02.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 shutdown_connections 2024-08-19T01:25:02.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1ac4039060 0x7f1ac403b520 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:02.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 --2- 172.21.15.161:0/100309777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ad4101e60 0x7f1ad4196ab0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:02.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 >> 172.21.15.161:0/100309777 conn(0x7f1ad40fba00 msgr2=0x7f1ad40fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:02.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 shutdown_connections 2024-08-19T01:25:02.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:02.087+0000 7f1adb0db640 1 -- 172.21.15.161:0/100309777 wait complete. 2024-08-19T01:25:02.697 DEBUG:teuthology.orchestra.run.smithi123:mon.b> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.b.service 2024-08-19T01:25:02.699 DEBUG:teuthology.orchestra.run.smithi161:mon.c> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.c.service 2024-08-19T01:25:02.702 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T01:25:02.702 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T01:25:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:25:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:02.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:02.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:02.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:03 smithi064 ceph-mon[18515]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:03 smithi064 ceph-mon[18515]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:25:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:03 smithi064 ceph-mon[18515]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:25:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:03 smithi064 ceph-mon[18515]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:03 smithi064 ceph-mon[18515]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c;count:3 2024-08-19T01:25:03.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:03 smithi064 ceph-mon[18515]: Deploying daemon mon.c on smithi161 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.535+0000 7f15ffeae640 1 -- 172.21.15.161:0/2015800269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 msgr2=0x7f15f8100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.535+0000 7f15ffeae640 1 --2- 172.21.15.161:0/2015800269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8100800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009880 tx=0x7f15f402a5d0 comp rx=0 tx=0).stop 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.535+0000 7f15ffeae640 1 -- 172.21.15.161:0/2015800269 shutdown_connections 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.535+0000 7f15ffeae640 1 --2- 172.21.15.161:0/2015800269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8100800 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.535+0000 7f15ffeae640 1 -- 172.21.15.161:0/2015800269 >> 172.21.15.161:0/2015800269 conn(0x7f15f80fbee0 msgr2=0x7f15f80fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.535+0000 7f15ffeae640 1 -- 172.21.15.161:0/2015800269 shutdown_connections 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.535+0000 7f15ffeae640 1 -- 172.21.15.161:0/2015800269 wait complete. 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15ffeae640 1 Processor -- start 2024-08-19T01:25:04.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15ffeae640 1 -- start start 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15ffeae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15ffeae640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15f810a870 con 0x7f15f8100420 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15fdc4a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15fdc4a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8195b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.161:45752/0 (socket says 172.21.15.161:45752) 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15fdc4a640 1 -- 172.21.15.161:0/2587605531 learned_addr learned my addr 172.21.15.161:0/2587605531 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.536+0000 7f15fdc4a640 1 -- 172.21.15.161:0/2587605531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f806aa00 con 0x7f15f8100420 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.537+0000 7f15fdc4a640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8195b00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009230 tx=0x7f15f4003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.537+0000 7f15eeffd640 1 -- 172.21.15.161:0/2587605531 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15f403f070 con 0x7f15f8100420 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.538+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f8196040 con 0x7f15f8100420 2024-08-19T01:25:04.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.538+0000 7f15eeffd640 1 -- 172.21.15.161:0/2587605531 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f403bc80 con 0x7f15f8100420 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.538+0000 7f15eeffd640 1 -- 172.21.15.161:0/2587605531 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15f4035030 con 0x7f15f8100420 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.538+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f8196500 con 0x7f15f8100420 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.538+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f8101900 con 0x7f15f8100420 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.538+0000 7f15eeffd640 1 -- 172.21.15.161:0/2587605531 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f15f4042070 con 0x7f15f8100420 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.539+0000 7f15eeffd640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f15e4039060 0x7f15e403b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.539+0000 7f15eeffd640 1 -- 172.21.15.161:0/2587605531 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f15f406cc90 con 0x7f15f8100420 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.539+0000 7f15fd449640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f15e4039060 0x7f15e403b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.539+0000 7f15fd449640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f15e4039060 0x7f15e403b520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f15e80098c0 tx=0x7f15e8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:04.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.544+0000 7f15eeffd640 1 -- 172.21.15.161:0/2587605531 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f15f4031e20 con 0x7f15f8100420 2024-08-19T01:25:04.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.778+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f15f806a1f0 con 0x7f15f8100420 2024-08-19T01:25:04.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.779+0000 7f15eeffd640 1 -- 172.21.15.161:0/2587605531 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f15f40372d0 con 0x7f15f8100420 2024-08-19T01:25:04.781 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:25:04.781 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":1,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","modified":"2024-08-19T01:23:45.058123Z","created":"2024-08-19T01:23:45.058123Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-19T01:25:04.781 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 1 2024-08-19T01:25:04.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f15e4039060 msgr2=0x7f15e403b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:04.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f15e4039060 0x7f15e403b520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f15e80098c0 tx=0x7f15e8006db0 comp rx=0 tx=0).stop 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 msgr2=0x7f15f8195b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8195b00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009230 tx=0x7f15f4003d40 comp rx=0 tx=0).stop 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 shutdown_connections 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f15e4039060 0x7f15e403b520 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 --2- 172.21.15.161:0/2587605531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15f8100420 0x7f15f8195b00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 >> 172.21.15.161:0/2587605531 conn(0x7f15f80fbee0 msgr2=0x7f15f8107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 shutdown_connections 2024-08-19T01:25:04.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:04.781+0000 7f15ffeae640 1 -- 172.21.15.161:0/2587605531 wait complete. 2024-08-19T01:25:05.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:05 smithi064 ceph-mon[18515]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:05.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:05 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/2587605531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.arena_block_size: 1048576 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.disable_auto_compactions: 0 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-19T01:25:06.420 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.inplace_update_support: 0 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-19T01:25:06.421 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.bloom_locality: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.max_successive_merges: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.paranoid_file_checks: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.force_consistency_checks: 1 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.report_bg_io_stats: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.ttl: 2592000 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.enable_blob_files: false 2024-08-19T01:25:06.422 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.min_blob_size: 0 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.blob_file_size: 268435456 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.blob_file_starting_level: 0 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 30a200e0-f895-481f-b306-b476d9bbca9b 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724030706416123, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-08-19T01:25:06.423 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724030706416621, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1724030706, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "30a200e0-f895-481f-b306-b476d9bbca9b", "db_session_id": "H0VEK6X8CEOT7TF1A1MC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724030706416741, "job": 1, "event": "recovery_finished"} 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560625e68e00 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: DB pointer 0x560625f62000 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** DB Stats ** 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-19T01:25:06.424 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** Compaction Stats [default] ** 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** Compaction Stats [default] ** 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T01:25:06.425 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Block cache BinnedLRUCache@0x560625e3b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-08-19T01:25:06.426 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-08-19T01:25:06.427 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-19T01:25:06.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-08-19T01:25:06.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: using public_addr v2:172.21.15.161:0/0 -> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-08-19T01:25:06.564 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T01:25:06.564 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: starting mon.c rank -1 at public addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] at bind addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(???) e0 preinit fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).mds e1 new map 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).mds e1 print_map 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout: e1 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout: btime 2024-08-19T01:23:48:740524+0000 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-19T01:25:06.700 INFO:journalctl@ceph.mon.c.smithi161.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout: legacy client fscid: -1 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout: No filesystems configured 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T01:25:06.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: Added host smithi161 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3681431991' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3681431991' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:25:06.703 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c;count:3 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: Deploying daemon mon.c on smithi161 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/2587605531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 ceph-mon[21757]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 bash[21737]: 425a47cf752d9afa6b4f2e095f21918f7afacf5943aedfd11c91eafc630373fe 2024-08-19T01:25:06.704 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:06 smithi161 systemd[1]: Started Ceph mon.c for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:25:06.810 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:25:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.613+0000 7f22bb790640 1 -- 172.21.15.161:0/2312244701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 msgr2=0x7f22a00059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.613+0000 7f22bb790640 1 --2- 172.21.15.161:0/2312244701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 0x7f22a00059c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f22b4101740 tx=0x7f22b002c490 comp rx=0 tx=0).stop 2024-08-19T01:25:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.613+0000 7f22bb790640 1 -- 172.21.15.161:0/2312244701 shutdown_connections 2024-08-19T01:25:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.613+0000 7f22bb790640 1 --2- 172.21.15.161:0/2312244701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 0x7f22a00059c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.613+0000 7f22bb790640 1 -- 172.21.15.161:0/2312244701 >> 172.21.15.161:0/2312244701 conn(0x7f22b40fbfb0 msgr2=0x7f22b40fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.613+0000 7f22bb790640 1 -- 172.21.15.161:0/2312244701 shutdown_connections 2024-08-19T01:25:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.613+0000 7f22bb790640 1 -- 172.21.15.161:0/2312244701 wait complete. 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.614+0000 7f22bb790640 1 Processor -- start 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.614+0000 7f22bb790640 1 -- start start 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.614+0000 7f22bb790640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 0x7f22b41970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22bb790640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 0x7f22b419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22bb790640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41083c0 con 0x7f22a0003530 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22bb790640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22b4108240 con 0x7f22b4197620 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b8d2b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 0x7f22b419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b8d2b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 0x7f22b419b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55928/0 (socket says 172.21.15.161:55928) 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b8d2b640 1 -- 172.21.15.161:0/2107121114 learned_addr learned my addr 172.21.15.161:0/2107121114 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b8d2b640 1 -- 172.21.15.161:0/2107121114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 msgr2=0x7f22b419b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b8d2b640 1 -- 172.21.15.161:0/2107121114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 msgr2=0x7f22b419b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b952c640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 0x7f22b41970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b8d2b640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 0x7f22b419b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-19T01:25:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b8d2b640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 0x7f22b419b910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:25:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b952c640 1 -- 172.21.15.161:0/2107121114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 msgr2=0x7f22b419b910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:25:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b952c640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 0x7f22b419b910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.615+0000 7f22b952c640 1 -- 172.21.15.161:0/2107121114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b4197c40 con 0x7f22a0003530 2024-08-19T01:25:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.616+0000 7f22b952c640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 0x7f22b41970e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f22b0000c00 tx=0x7f22b0032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.616+0000 7f22aa7fc640 1 -- 172.21.15.161:0/2107121114 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f22b0038040 con 0x7f22a0003530 2024-08-19T01:25:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.616+0000 7f22aa7fc640 1 -- 172.21.15.161:0/2107121114 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22b0005040 con 0x7f22a0003530 2024-08-19T01:25:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.616+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b419be50 con 0x7f22a0003530 2024-08-19T01:25:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.616+0000 7f22aa7fc640 1 -- 172.21.15.161:0/2107121114 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f22b00049a0 con 0x7f22a0003530 2024-08-19T01:25:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.616+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b419c390 con 0x7f22a0003530 2024-08-19T01:25:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.617+0000 7f22aa7fc640 1 -- 172.21.15.161:0/2107121114 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f22b0043020 con 0x7f22a0003530 2024-08-19T01:25:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.617+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b41019f0 con 0x7f22a0003530 2024-08-19T01:25:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.617+0000 7f22aa7fc640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2298038e50 0x7f229803b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.617+0000 7f22aa7fc640 1 -- 172.21.15.161:0/2107121114 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f22b006e590 con 0x7f22a0003530 2024-08-19T01:25:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.618+0000 7f22b8d2b640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2298038e50 0x7f229803b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.618+0000 7f22b8d2b640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2298038e50 0x7f229803b310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009100 tx=0x7f22a4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.623+0000 7f22aa7fc640 1 -- 172.21.15.161:0/2107121114 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f22b004a020 con 0x7f22a0003530 2024-08-19T01:25:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: mon.a calling monitor election 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: mon.c calling monitor election 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: monmap epoch 2 2024-08-19T01:25:11.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: last_changed 2024-08-19T01:25:06.443611+0000 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: created 2024-08-19T01:23:45.058123+0000 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: min_mon_release 19 (squid) 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: election_strategy: 1 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: fsmap 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: mgrmap e13: x(active, since 37s) 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: overall HEALTH_OK 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:11.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:11.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:11.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:11.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.853+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f22b419c6d0 con 0x7f22a0003530 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: mon.a calling monitor election 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: mon.c calling monitor election 2024-08-19T01:25:11.856 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: monmap epoch 2 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: last_changed 2024-08-19T01:25:06.443611+0000 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: created 2024-08-19T01:23:45.058123+0000 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: min_mon_release 19 (squid) 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: election_strategy: 1 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: fsmap 2024-08-19T01:25:11.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T01:25:11.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: mgrmap e13: x(active, since 37s) 2024-08-19T01:25:11.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: overall HEALTH_OK 2024-08-19T01:25:11.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:11.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:11.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:11.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:11.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.854+0000 7f22aa7fc640 1 -- 172.21.15.161:0/2107121114 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f22b00055a0 con 0x7f22a0003530 2024-08-19T01:25:11.858 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:25:11.859 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":2,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","modified":"2024-08-19T01:25:06.443611Z","created":"2024-08-19T01:23:45.058123Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-19T01:25:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 2 2024-08-19T01:25:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2298038e50 msgr2=0x7f229803b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2298038e50 0x7f229803b310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009100 tx=0x7f22a4008040 comp rx=0 tx=0).stop 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 msgr2=0x7f22b41970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 0x7f22b41970e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f22b0000c00 tx=0x7f22b0032bc0 comp rx=0 tx=0).stop 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 shutdown_connections 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2298038e50 0x7f229803b310 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b4197620 0x7f22b419b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 --2- 172.21.15.161:0/2107121114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0003530 0x7f22b41970e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 >> 172.21.15.161:0/2107121114 conn(0x7f22b40fbfb0 msgr2=0x7f22b40fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 shutdown_connections 2024-08-19T01:25:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:11.856+0000 7f22bb790640 1 -- 172.21.15.161:0/2107121114 wait complete. 2024-08-19T01:25:12.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:12 smithi161 ceph-mon[21757]: Deploying daemon mon.b on smithi123 2024-08-19T01:25:12.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:12 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/2107121114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:12.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:12 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:12.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:12 smithi064 ceph-mon[18515]: Deploying daemon mon.b on smithi123 2024-08-19T01:25:12.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:12 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/2107121114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:12.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:12 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:13.436 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T01:25:13.437 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T01:25:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:25:13.801 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:13 smithi161 ceph-mon[21757]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:13.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:13 smithi064 ceph-mon[18515]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:14.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.047+0000 7f1fceef7640 1 -- 172.21.15.161:0/4088350551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 msgr2=0x7f1fc81008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.047+0000 7f1fceef7640 1 --2- 172.21.15.161:0/4088350551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 0x7f1fc81008d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009880 tx=0x7f1fb802a5d0 comp rx=0 tx=0).stop 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.047+0000 7f1fceef7640 1 -- 172.21.15.161:0/4088350551 shutdown_connections 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.047+0000 7f1fceef7640 1 --2- 172.21.15.161:0/4088350551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 0x7f1fc81008d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.047+0000 7f1fceef7640 1 -- 172.21.15.161:0/4088350551 >> 172.21.15.161:0/4088350551 conn(0x7f1fc80fbf50 msgr2=0x7f1fc80fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.047+0000 7f1fceef7640 1 -- 172.21.15.161:0/4088350551 shutdown_connections 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.047+0000 7f1fceef7640 1 -- 172.21.15.161:0/4088350551 wait complete. 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fceef7640 1 Processor -- start 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fceef7640 1 -- start start 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fceef7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 0x7f1fc8196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fceef7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 0x7f1fc819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fceef7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc810a940 con 0x7f1fc81004f0 2024-08-19T01:25:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fceef7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc810a7c0 con 0x7f1fc8196580 2024-08-19T01:25:14.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fc7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 0x7f1fc819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:14.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fc7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 0x7f1fc819a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55940/0 (socket says 172.21.15.161:55940) 2024-08-19T01:25:14.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fc7fff640 1 -- 172.21.15.161:0/2196742808 learned_addr learned my addr 172.21.15.161:0/2196742808 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:25:14.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fccc93640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 0x7f1fc8196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:14.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fc7fff640 1 -- 172.21.15.161:0/2196742808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 msgr2=0x7f1fc819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T01:25:14.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fc7fff640 1 -- 172.21.15.161:0/2196742808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 msgr2=0x7f1fc819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T01:25:14.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fc7fff640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 0x7f1fc819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fc7fff640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 0x7f1fc819a8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fccc93640 1 -- 172.21.15.161:0/2196742808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 msgr2=0x7f1fc819a8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fccc93640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 0x7f1fc819a8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.048+0000 7f1fccc93640 1 -- 172.21.15.161:0/2196742808 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc8196ba0 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.049+0000 7f1fccc93640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 0x7f1fc8196040 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8005ab0 tx=0x7f1fb8032c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.049+0000 7f1fc5ffb640 1 -- 172.21.15.161:0/2196742808 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1fb8035070 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.049+0000 7f1fc5ffb640 1 -- 172.21.15.161:0/2196742808 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fb8032e40 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.049+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc819ae00 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.049+0000 7f1fc5ffb640 1 -- 172.21.15.161:0/2196742808 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1fb803bc10 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.049+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc819b340 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.050+0000 7f1fc5ffb640 1 -- 172.21.15.161:0/2196742808 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1fb8042030 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.050+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc81019d0 con 0x7f1fc81004f0 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.050+0000 7f1fc5ffb640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1fb0038e00 0x7f1fb003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:14.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.050+0000 7f1fc5ffb640 1 -- 172.21.15.161:0/2196742808 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1fb8040070 con 0x7f1fc81004f0 2024-08-19T01:25:14.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.051+0000 7f1fc7fff640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1fb0038e00 0x7f1fb003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:14.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.051+0000 7f1fc7fff640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1fb0038e00 0x7f1fb003b2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009100 tx=0x7f1fbc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:14.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.055+0000 7f1fc5ffb640 1 -- 172.21.15.161:0/2196742808 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1fb803bdb0 con 0x7f1fc81004f0 2024-08-19T01:25:14.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.285+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1fc819b620 con 0x7f1fc81004f0 2024-08-19T01:25:14.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.285+0000 7f1fc5ffb640 1 -- 172.21.15.161:0/2196742808 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f1fb803b460 con 0x7f1fc81004f0 2024-08-19T01:25:14.287 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:25:14.288 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":2,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","modified":"2024-08-19T01:25:06.443611Z","created":"2024-08-19T01:23:45.058123Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-19T01:25:14.288 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 2 2024-08-19T01:25:14.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1fb0038e00 msgr2=0x7f1fb003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:14.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1fb0038e00 0x7f1fb003b2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009100 tx=0x7f1fbc008040 comp rx=0 tx=0).stop 2024-08-19T01:25:14.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 msgr2=0x7f1fc8196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:14.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 0x7f1fc8196040 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8005ab0 tx=0x7f1fb8032c50 comp rx=0 tx=0).stop 2024-08-19T01:25:14.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 shutdown_connections 2024-08-19T01:25:14.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1fb0038e00 0x7f1fb003b2c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:14.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc8196580 0x7f1fc819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:14.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 --2- 172.21.15.161:0/2196742808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81004f0 0x7f1fc8196040 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:14.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 >> 172.21.15.161:0/2196742808 conn(0x7f1fc80fbf50 msgr2=0x7f1fc8107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:14.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 shutdown_connections 2024-08-19T01:25:14.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:14.287+0000 7f1fceef7640 1 -- 172.21.15.161:0/2196742808 wait complete. 2024-08-19T01:25:14.804 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:14 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/2196742808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:14.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:14 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/2196742808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:15.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:15 smithi123 systemd[1]: Started Ceph mon.b for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:25:15.911 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T01:25:15.911 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T01:25:16.162 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: mon.a calling monitor election 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: mon.c calling monitor election 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: mon.b calling monitor election 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: monmap epoch 3 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: last_changed 2024-08-19T01:25:15.279855+0000 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: created 2024-08-19T01:23:45.058123+0000 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: min_mon_release 19 (squid) 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: election_strategy: 1 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: 2: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: fsmap 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T01:25:20.668 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: mgrmap e13: x(active, since 46s) 2024-08-19T01:25:20.669 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: overall HEALTH_OK 2024-08-19T01:25:20.669 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.669 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.669 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.669 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: mon.a calling monitor election 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: mon.c calling monitor election 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: mon.b calling monitor election 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: monmap epoch 3 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: last_changed 2024-08-19T01:25:15.279855+0000 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: created 2024-08-19T01:23:45.058123+0000 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: min_mon_release 19 (squid) 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: election_strategy: 1 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: 2: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: fsmap 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: mgrmap e13: x(active, since 46s) 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: overall HEALTH_OK 2024-08-19T01:25:20.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:25:20.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: mon.a calling monitor election 2024-08-19T01:25:20.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: mon.c calling monitor election 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: mon.b calling monitor election 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: monmap epoch 3 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:20.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: last_changed 2024-08-19T01:25:15.279855+0000 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: created 2024-08-19T01:23:45.058123+0000 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: min_mon_release 19 (squid) 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: election_strategy: 1 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: 2: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: fsmap 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: mgrmap e13: x(active, since 46s) 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: overall HEALTH_OK 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.974+0000 7f23e5052640 1 -- 172.21.15.161:0/876363091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 msgr2=0x7f23cc0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.974+0000 7f23e5052640 1 --2- 172.21.15.161:0/876363091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 0x7f23cc0059c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23e006a6d0 tx=0x7f23d002c490 comp rx=0 tx=0).stop 2024-08-19T01:25:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.974+0000 7f23e5052640 1 -- 172.21.15.161:0/876363091 shutdown_connections 2024-08-19T01:25:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.974+0000 7f23e5052640 1 --2- 172.21.15.161:0/876363091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 0x7f23cc0059c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.974+0000 7f23e5052640 1 -- 172.21.15.161:0/876363091 >> 172.21.15.161:0/876363091 conn(0x7f23e00fbf50 msgr2=0x7f23e00fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.974+0000 7f23e5052640 1 -- 172.21.15.161:0/876363091 shutdown_connections 2024-08-19T01:25:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.974+0000 7f23e5052640 1 -- 172.21.15.161:0/876363091 wait complete. 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.975+0000 7f23e5052640 1 Processor -- start 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23e5052640 1 -- start start 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23e5052640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 0x7f23e0197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23e5052640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 0x7f23e019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23e5052640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23e019bf50 0x7f23e019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23e5052640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f23e0109470 con 0x7f23cc003530 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23e5052640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01092f0 con 0x7f23e019bf50 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23e5052640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23e01095f0 con 0x7f23e0197680 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23de59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 0x7f23e019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23de59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 0x7f23e019ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40794/0 (socket says 172.21.15.161:40794) 2024-08-19T01:25:20.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23de59c640 1 -- 172.21.15.161:0/2054848963 learned_addr learned my addr 172.21.15.161:0/2054848963 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23ded9d640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 0x7f23e0197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23de59c640 1 -- 172.21.15.161:0/2054848963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 msgr2=0x7f23e019ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23de59c640 1 -- 172.21.15.161:0/2054848963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 msgr2=0x7f23e019ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23de59c640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 0x7f23e019ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23de59c640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 0x7f23e019ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23ded9d640 1 -- 172.21.15.161:0/2054848963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 msgr2=0x7f23e019ba10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23ded9d640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 0x7f23e019ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23df59e640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23e019bf50 0x7f23e019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23ded9d640 1 -- 172.21.15.161:0/2054848963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23e019bf50 msgr2=0x7f23e019c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23ded9d640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23e019bf50 0x7f23e019c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23ded9d640 1 -- 172.21.15.161:0/2054848963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e01a3cc0 con 0x7f23cc003530 2024-08-19T01:25:20.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.976+0000 7f23df59e640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23e019bf50 0x7f23e019c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.977+0000 7f23ded9d640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 0x7f23e0197140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23d0005f30 tx=0x7f23d00337f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.977+0000 7f23d7fff640 1 -- 172.21.15.161:0/2054848963 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0040020 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.977+0000 7f23d7fff640 1 -- 172.21.15.161:0/2054848963 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d003c410 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.977+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e01a3e90 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.977+0000 7f23d7fff640 1 -- 172.21.15.161:0/2054848963 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0038040 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.977+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e01a42e0 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.978+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e010a600 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.978+0000 7f23d7fff640 1 -- 172.21.15.161:0/2054848963 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f23d0043020 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.978+0000 7f23d7fff640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f23c00394f0 0x7f23c003b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.978+0000 7f23d7fff640 1 -- 172.21.15.161:0/2054848963 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f23d006e5e0 con 0x7f23cc003530 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.979+0000 7f23de59c640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f23c00394f0 0x7f23c003b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:20.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.979+0000 7f23de59c640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f23c00394f0 0x7f23c003b9b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009100 tx=0x7f23c8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:20.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:20.983+0000 7f23d7fff640 1 -- 172.21.15.161:0/2054848963 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f23d004a020 con 0x7f23cc003530 2024-08-19T01:25:21.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.223+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f23e01a4740 con 0x7f23cc003530 2024-08-19T01:25:21.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.223+0000 7f23d7fff640 1 -- 172.21.15.161:0/2054848963 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f23d0043310 con 0x7f23cc003530 2024-08-19T01:25:21.226 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:25:21.226 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":3,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","modified":"2024-08-19T01:25:15.279855Z","created":"2024-08-19T01:23:45.058123Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-08-19T01:25:21.226 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 3 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f23c00394f0 msgr2=0x7f23c003b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f23c00394f0 0x7f23c003b9b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009100 tx=0x7f23c8008040 comp rx=0 tx=0).stop 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 msgr2=0x7f23e0197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 0x7f23e0197140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23d0005f30 tx=0x7f23d00337f0 comp rx=0 tx=0).stop 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 shutdown_connections 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f23c00394f0 0x7f23c003b9b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23e019bf50 0x7f23e019c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23e0197680 0x7f23e019ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 --2- 172.21.15.161:0/2054848963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23cc003530 0x7f23e0197140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.225+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 >> 172.21.15.161:0/2054848963 conn(0x7f23e00fbf50 msgr2=0x7f23e00fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.226+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 shutdown_connections 2024-08-19T01:25:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:21.226+0000 7f23e5052640 1 -- 172.21.15.161:0/2054848963 wait complete. 2024-08-19T01:25:21.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:21 smithi123 ceph-mon[21829]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:21.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:21 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:21.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:21 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/2054848963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:21.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:21 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:21.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:21 smithi161 ceph-mon[21757]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:21.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:21 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:21.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:21 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/2054848963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:21.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:21 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:21.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:21 smithi064 ceph-mon[18515]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:21.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:21 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:21.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:21 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/2054848963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T01:25:21.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:21 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:25:22.120 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-08-19T01:25:22.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph config generate-minimal-conf 2024-08-19T01:25:22.405 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:25:22.280+0000 7f70da9f2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: Updating smithi064:/etc/ceph/ceph.conf 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: Updating smithi123:/etc/ceph/ceph.conf 2024-08-19T01:25:22.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:22.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:22.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: Updating smithi064:/etc/ceph/ceph.conf 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: Updating smithi123:/etc/ceph/ceph.conf 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:22.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:22.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: Updating smithi064:/etc/ceph/ceph.conf 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: Updating smithi123:/etc/ceph/ceph.conf 2024-08-19T01:25:22.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:22.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:23.373 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:25:23.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:23.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: Reconfiguring mon.a (unknown last config time)... 2024-08-19T01:25:23.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: Reconfiguring daemon mon.a on smithi064 2024-08-19T01:25:23.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:23.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:23.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:23.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:23.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:23 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:23.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: Reconfiguring mon.a (unknown last config time)... 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: Reconfiguring daemon mon.a on smithi064 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:23 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:23.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:23.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:25:23.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: Reconfiguring mon.a (unknown last config time)... 2024-08-19T01:25:23.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: Reconfiguring daemon mon.a on smithi064 2024-08-19T01:25:23.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:23.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:23.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:23.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:23.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:23 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 -- 172.21.15.64:0/663336860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c1022a0 msgr2=0x7f877c1026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 --2- 172.21.15.64:0/663336860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c1022a0 0x7f877c1026e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f876c009900 tx=0x7f876c02a480 comp rx=0 tx=0).stop 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 -- 172.21.15.64:0/663336860 shutdown_connections 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 --2- 172.21.15.64:0/663336860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f877c102c20 0x7f877c10d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 --2- 172.21.15.64:0/663336860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c1022a0 0x7f877c1026e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 --2- 172.21.15.64:0/663336860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f877c100b50 0x7f877c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 -- 172.21.15.64:0/663336860 >> 172.21.15.64:0/663336860 conn(0x7f877c0fc500 msgr2=0x7f877c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 -- 172.21.15.64:0/663336860 shutdown_connections 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.808+0000 7f87836cc640 1 -- 172.21.15.64:0/663336860 wait complete. 2024-08-19T01:25:23.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 Processor -- start 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 -- start start 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f877c100b50 0x7f877c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f877c1022a0 0x7f877c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c102c20 0x7f877c19c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f877c110cb0 con 0x7f877c102c20 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f877c110b30 con 0x7f877c100b50 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f87836cc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f877c110e30 con 0x7f877c1022a0 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c102c20 0x7f877c19c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c102c20 0x7f877c19c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41156/0 (socket says 172.21.15.64:41156) 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 -- 172.21.15.64:0/1196847797 learned_addr learned my addr 172.21.15.64:0/1196847797 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781468640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f877c100b50 0x7f877c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8780c67640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f877c1022a0 0x7f877c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:23.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 -- 172.21.15.64:0/1196847797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f877c1022a0 msgr2=0x7f877c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f877c1022a0 0x7f877c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 -- 172.21.15.64:0/1196847797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f877c100b50 msgr2=0x7f877c198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f877c100b50 0x7f877c198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8781c69640 1 -- 172.21.15.64:0/1196847797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877c19d0d0 con 0x7f877c102c20 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.809+0000 7f8780c67640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f877c1022a0 0x7f877c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.810+0000 7f8781c69640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c102c20 0x7f877c19c9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f877400cce0 tx=0x7f877400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.810+0000 7f87727fc640 1 -- 172.21.15.64:0/1196847797 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877400f450 con 0x7f877c102c20 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.810+0000 7f87727fc640 1 -- 172.21.15.64:0/1196847797 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f877400fad0 con 0x7f877c102c20 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.810+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877c19d3c0 con 0x7f877c102c20 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.810+0000 7f87727fc640 1 -- 172.21.15.64:0/1196847797 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774018b60 con 0x7f877c102c20 2024-08-19T01:25:23.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.810+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f877c1a4ca0 con 0x7f877c102c20 2024-08-19T01:25:23.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.811+0000 7f87727fc640 1 -- 172.21.15.64:0/1196847797 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f87740213f0 con 0x7f877c102c20 2024-08-19T01:25:23.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.811+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877c100f30 con 0x7f877c102c20 2024-08-19T01:25:23.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.811+0000 7f87727fc640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8768039290 0x7f876803b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:23.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.811+0000 7f87727fc640 1 -- 172.21.15.64:0/1196847797 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f877404c8a0 con 0x7f877c102c20 2024-08-19T01:25:23.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.811+0000 7f8781468640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8768039290 0x7f876803b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:23.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.811+0000 7f8781468640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8768039290 0x7f876803b750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f877800abe0 tx=0x7f877800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:23.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.816+0000 7f87727fc640 1 -- 172.21.15.64:0/1196847797 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f8774014030 con 0x7f877c102c20 2024-08-19T01:25:23.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.992+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f877c109e00 con 0x7f877c102c20 2024-08-19T01:25:23.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.993+0000 7f87727fc640 1 -- 172.21.15.64:0/1196847797 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f877400fcf0 con 0x7f877c102c20 2024-08-19T01:25:23.995 INFO:teuthology.orchestra.run.smithi064.stdout:# minimal ceph.conf for 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:23.995 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-08-19T01:25:23.995 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:25:23.995 INFO:teuthology.orchestra.run.smithi064.stdout: mon_host = [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-08-19T01:25:23.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8768039290 msgr2=0x7f876803b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:23.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8768039290 0x7f876803b750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f877800abe0 tx=0x7f877800a2d0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c102c20 msgr2=0x7f877c19c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:23.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c102c20 0x7f877c19c9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f877400cce0 tx=0x7f877400a620 comp rx=0 tx=0).stop 2024-08-19T01:25:23.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 shutdown_connections 2024-08-19T01:25:23.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8768039290 0x7f876803b750 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f877c102c20 0x7f877c19c9f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f877c1022a0 0x7f877c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 --2- 172.21.15.64:0/1196847797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f877c100b50 0x7f877c198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:23.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 >> 172.21.15.64:0/1196847797 conn(0x7f877c0fc500 msgr2=0x7f877c069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:23.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 shutdown_connections 2024-08-19T01:25:23.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:23.995+0000 7f87836cc640 1 -- 172.21.15.64:0/1196847797 wait complete. 2024-08-19T01:25:24.541 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:24 smithi064 ceph-mon[18515]: Reconfiguring mon.b (monmap changed)... 2024-08-19T01:25:24.541 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:24 smithi064 ceph-mon[18515]: Reconfiguring daemon mon.b on smithi123 2024-08-19T01:25:24.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:24.542 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:24 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1196847797' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:24.569 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-08-19T01:25:24.569 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:25:24.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-08-19T01:25:24.596 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:25:24.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:25:24.661 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:25:24.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2024-08-19T01:25:24.689 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:24 smithi123 ceph-mon[21829]: Reconfiguring mon.b (monmap changed)... 2024-08-19T01:25:24.690 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:24 smithi123 ceph-mon[21829]: Reconfiguring daemon mon.b on smithi123 2024-08-19T01:25:24.690 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:24.690 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:24 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1196847797' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:24.692 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:25:24.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:25:24.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:24 smithi161 ceph-mon[21757]: Reconfiguring mon.b (monmap changed)... 2024-08-19T01:25:24.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:24 smithi161 ceph-mon[21757]: Reconfiguring daemon mon.b on smithi123 2024-08-19T01:25:24.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:24.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:24 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1196847797' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:24.762 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:25:24.762 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.conf 2024-08-19T01:25:24.789 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:25:24.789 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:25:24.855 INFO:tasks.cephadm:Adding mgr.x on smithi064 2024-08-19T01:25:24.855 INFO:tasks.cephadm:Adding mgr.y on smithi123 2024-08-19T01:25:24.855 INFO:tasks.cephadm:Adding mgr.z on smithi161 2024-08-19T01:25:24.855 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch apply mgr '3;smithi064=x;smithi123=y;smithi161=z' 2024-08-19T01:25:25.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:25 smithi064 ceph-mon[18515]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:25.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:25 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:25.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:25 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:25.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:25 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:25.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:25 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:25.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:25 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:25.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:25 smithi161 ceph-mon[21757]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:25.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:25 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:25.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:25 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:25.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:25 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:25.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:25 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:25.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:25 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:25.707 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:25:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:25 smithi123 ceph-mon[21829]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:25 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:25 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:25.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:25 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T01:25:25.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:25 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T01:25:25.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:25 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:26.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 -- 172.21.15.161:0/1926904005 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 msgr2=0x7f8400100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:26.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 --2- 172.21.15.161:0/1926904005 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f8400100720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009880 tx=0x7f83fc02a5d0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 -- 172.21.15.161:0/1926904005 shutdown_connections 2024-08-19T01:25:26.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 --2- 172.21.15.161:0/1926904005 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f8400100720 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 -- 172.21.15.161:0/1926904005 >> 172.21.15.161:0/1926904005 conn(0x7f84000fbad0 msgr2=0x7f84000fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:26.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 -- 172.21.15.161:0/1926904005 shutdown_connections 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 -- 172.21.15.161:0/1926904005 wait complete. 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 Processor -- start 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.164+0000 7f8408340640 1 -- start start 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f8408340640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f840006ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f8408340640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840019be20 0x7f840019e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f8408340640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f840019e750 0x7f84001a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f8408340640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8400106410 con 0x7f8400100320 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f8408340640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8400106290 con 0x7f840019e750 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f8408340640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8400106590 con 0x7f840019be20 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840019be20 0x7f840019e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84060dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f840006ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840019be20 0x7f840019e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57044/0 (socket says 172.21.15.161:57044) 2024-08-19T01:25:26.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84060dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f840006ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.161:41392/0 (socket says 172.21.15.161:41392) 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 -- 172.21.15.161:0/374998614 learned_addr learned my addr 172.21.15.161:0/374998614 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84068dd640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f840019e750 0x7f84001a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 -- 172.21.15.161:0/374998614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f840019e750 msgr2=0x7f84001a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f840019e750 0x7f84001a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 -- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 msgr2=0x7f840006ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f840006ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 -- 172.21.15.161:0/374998614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84001a1310 con 0x7f840019be20 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84060dc640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f840006ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84068dd640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f840019e750 0x7f84001a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.165+0000 7f84058db640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840019be20 0x7f840019e210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f83f400b520 tx=0x7f83f400b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.166+0000 7f83f37fe640 1 -- 172.21.15.161:0/374998614 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4004230 con 0x7f840019be20 2024-08-19T01:25:26.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.166+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84001a1600 con 0x7f840019be20 2024-08-19T01:25:26.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.166+0000 7f83f37fe640 1 -- 172.21.15.161:0/374998614 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83f40043d0 con 0x7f840019be20 2024-08-19T01:25:26.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.166+0000 7f83f37fe640 1 -- 172.21.15.161:0/374998614 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4003d60 con 0x7f840019be20 2024-08-19T01:25:26.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.166+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001a1c10 con 0x7f840019be20 2024-08-19T01:25:26.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.167+0000 7f83f37fe640 1 -- 172.21.15.161:0/374998614 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f83f400e070 con 0x7f840019be20 2024-08-19T01:25:26.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.167+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f840018f1d0 con 0x7f840019be20 2024-08-19T01:25:26.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.167+0000 7f83f37fe640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f83ec0394f0 0x7f83ec03b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:26.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.167+0000 7f83f37fe640 1 -- 172.21.15.161:0/374998614 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f83f404bca0 con 0x7f840019be20 2024-08-19T01:25:26.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.167+0000 7f84060dc640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f83ec0394f0 0x7f83ec03b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:26.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.168+0000 7f84060dc640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f83ec0394f0 0x7f83ec03b9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0314e0 tx=0x7f83fc0315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:26.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.172+0000 7f83f37fe640 1 -- 172.21.15.161:0/374998614 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f83f4014020 con 0x7f840019be20 2024-08-19T01:25:26.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.354+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi123=y;smithi161=z", "target": ["mon-mgr", ""]}) -- 0x7f84000623f0 con 0x7f83ec0394f0 2024-08-19T01:25:26.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.361+0000 7f83f37fe640 1 -- 172.21.15.161:0/374998614 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f84000623f0 con 0x7f83ec0394f0 2024-08-19T01:25:26.363 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mgr update... 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f83ec0394f0 msgr2=0x7f83ec03b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f83ec0394f0 0x7f83ec03b9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0314e0 tx=0x7f83fc0315d0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840019be20 msgr2=0x7f840019e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840019be20 0x7f840019e210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f83f400b520 tx=0x7f83f400b9e0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 shutdown_connections 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f83ec0394f0 0x7f83ec03b9b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f840019e750 0x7f84001a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840019be20 0x7f840019e210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 --2- 172.21.15.161:0/374998614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8400100320 0x7f840006ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:26.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 >> 172.21.15.161:0/374998614 conn(0x7f84000fbad0 msgr2=0x7f84000fc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:26.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 shutdown_connections 2024-08-19T01:25:26.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:25:26.363+0000 7f8408340640 1 -- 172.21.15.161:0/374998614 wait complete. 2024-08-19T01:25:26.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:26 smithi161 ceph-mon[21757]: Reconfiguring mon.c (monmap changed)... 2024-08-19T01:25:27.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:26 smithi161 ceph-mon[21757]: Reconfiguring daemon mon.c on smithi161 2024-08-19T01:25:27.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:26 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.035 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:26 smithi123 ceph-mon[21829]: Reconfiguring mon.c (monmap changed)... 2024-08-19T01:25:27.035 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:26 smithi123 ceph-mon[21829]: Reconfiguring daemon mon.c on smithi161 2024-08-19T01:25:27.036 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:26 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:26 smithi064 ceph-mon[18515]: Reconfiguring mon.c (monmap changed)... 2024-08-19T01:25:27.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:26 smithi064 ceph-mon[18515]: Reconfiguring daemon mon.c on smithi161 2024-08-19T01:25:27.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:26 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.256 DEBUG:teuthology.orchestra.run.smithi123:mgr.y> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.y.service 2024-08-19T01:25:27.259 DEBUG:teuthology.orchestra.run.smithi161:mgr.z> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.z.service 2024-08-19T01:25:27.261 INFO:tasks.cephadm:Deploying OSDs... 2024-08-19T01:25:27.262 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:25:27.262 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T01:25:27.282 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T01:25:27.282 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-08-19 01:24:42.263925371 +0000 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-08-19 01:20:32.895172256 +0000 2024-08-19T01:25:27.340 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-08-19 01:20:32.895172256 +0000 2024-08-19T01:25:27.341 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-08-19 01:20:32.895172256 +0000 2024-08-19T01:25:27.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T01:25:27.408 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-08-19T01:25:27.409 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-08-19T01:25:27.409 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000234594 s, 2.2 MB/s 2024-08-19T01:25:27.410 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T01:25:27.469 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-08-19T01:25:27.496 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:27.496 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi123=y;smithi161=z", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:27.497 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: Saving service mgr spec with placement smithi064=x;smithi123=y;smithi161=z;count:3 2024-08-19T01:25:27.497 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.497 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.497 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:27.497 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:27.497 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:27.497 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.498 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:27.498 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T01:25:27.498 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:27.498 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-08-19 01:24:42.291924827 +0000 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-08-19 01:20:33.223168166 +0000 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-08-19 01:20:33.223168166 +0000 2024-08-19T01:25:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-08-19 01:20:33.223168166 +0000 2024-08-19T01:25:27.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T01:25:27.590 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-08-19T01:25:27.591 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-08-19T01:25:27.591 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000195788 s, 2.6 MB/s 2024-08-19T01:25:27.592 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T01:25:27.650 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-08-19T01:25:27.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:27.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi123=y;smithi161=z", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:27.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: Saving service mgr spec with placement smithi064=x;smithi123=y;smithi161=z;count:3 2024-08-19T01:25:27.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:27.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:27.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:27.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:27.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T01:25:27.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:27.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-08-19 01:24:42.318924302 +0000 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-08-19 01:20:33.556164015 +0000 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-08-19 01:20:33.556164015 +0000 2024-08-19T01:25:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-08-19 01:20:33.556164015 +0000 2024-08-19T01:25:27.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi123=y;smithi161=z", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: Saving service mgr spec with placement smithi064=x;smithi123=y;smithi161=z;count:3 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:27.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:27.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:27.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T01:25:27.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:27.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:27.760 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-08-19T01:25:27.760 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-08-19T01:25:27.760 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000201073 s, 2.5 MB/s 2024-08-19T01:25:27.761 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T01:25:27.818 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-08-19T01:25:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-08-19 01:24:42.344923797 +0000 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-08-19 01:20:33.892159825 +0000 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-08-19 01:20:33.892159825 +0000 2024-08-19T01:25:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-08-19 01:20:33.892159825 +0000 2024-08-19T01:25:27.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T01:25:27.938 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-08-19T01:25:27.938 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-08-19T01:25:27.938 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000244562 s, 2.1 MB/s 2024-08-19T01:25:27.939 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T01:25:27.996 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-08-19T01:25:27.996 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T01:25:28.018 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T01:25:28.018 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-08-19 01:25:14.740726252 +0000 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-08-19 01:20:51.832063645 +0000 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-08-19 01:20:51.832063645 +0000 2024-08-19T01:25:28.074 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-08-19 01:20:51.832063645 +0000 2024-08-19T01:25:28.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T01:25:28.142 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-08-19T01:25:28.142 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-08-19T01:25:28.142 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000307677 s, 1.7 MB/s 2024-08-19T01:25:28.144 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T01:25:28.203 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 692 Links: 1 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-08-19 01:25:14.741726229 +0000 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-08-19 01:20:52.164056682 +0000 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-08-19 01:20:52.164056682 +0000 2024-08-19T01:25:28.260 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-08-19 01:20:52.164056682 +0000 2024-08-19T01:25:28.260 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T01:25:28.325 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-08-19T01:25:28.325 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-08-19T01:25:28.325 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000213314 s, 2.4 MB/s 2024-08-19T01:25:28.327 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T01:25:28.384 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 710 Links: 1 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-08-19 01:25:14.741726229 +0000 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-08-19 01:20:52.495049740 +0000 2024-08-19T01:25:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-08-19 01:20:52.495049740 +0000 2024-08-19T01:25:28.443 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-08-19 01:20:52.495049740 +0000 2024-08-19T01:25:28.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T01:25:28.506 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-08-19T01:25:28.506 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-08-19T01:25:28.506 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000224633 s, 2.3 MB/s 2024-08-19T01:25:28.507 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T01:25:28.564 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-08-19T01:25:28.620 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 730 Links: 1 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-08-19 01:25:14.741726229 +0000 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-08-19 01:20:52.822042881 +0000 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-08-19 01:20:52.822042881 +0000 2024-08-19T01:25:28.621 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-08-19 01:20:52.822042881 +0000 2024-08-19T01:25:28.621 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T01:25:28.686 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-08-19T01:25:28.686 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-08-19T01:25:28.686 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000249359 s, 2.1 MB/s 2024-08-19T01:25:28.687 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T01:25:28.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:28 smithi161 ceph-mon[21757]: Deploying daemon mgr.z on smithi161 2024-08-19T01:25:28.700 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:28 smithi161 systemd[1]: Starting Ceph mgr.z for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:25:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:28 smithi064 ceph-mon[18515]: Deploying daemon mgr.z on smithi161 2024-08-19T01:25:28.745 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-08-19T01:25:28.745 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T01:25:28.761 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T01:25:28.761 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-08-19T01:25:28.817 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T01:25:28.817 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:28.817 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 673 Links: 1 2024-08-19T01:25:28.817 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:28.817 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:28.817 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-08-19 01:25:28.541061581 +0000 2024-08-19T01:25:28.818 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-08-19 01:20:51.730134735 +0000 2024-08-19T01:25:28.818 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-08-19 01:20:51.730134735 +0000 2024-08-19T01:25:28.818 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-08-19 01:20:51.730134735 +0000 2024-08-19T01:25:28.818 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T01:25:28.883 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-08-19T01:25:28.884 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-08-19T01:25:28.884 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000245473 s, 2.1 MB/s 2024-08-19T01:25:28.885 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T01:25:28.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:28 smithi123 ceph-mon[21829]: Deploying daemon mgr.z on smithi161 2024-08-19T01:25:28.942 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-08-19T01:25:28.997 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T01:25:28.997 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:28.997 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-08-19T01:25:28.997 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:28.997 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:28.997 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-08-19 01:25:28.541061581 +0000 2024-08-19T01:25:28.997 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-08-19 01:20:52.058128877 +0000 2024-08-19T01:25:28.998 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-08-19 01:20:52.058128877 +0000 2024-08-19T01:25:28.998 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-08-19 01:20:52.058128877 +0000 2024-08-19T01:25:28.998 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T01:25:29.060 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-08-19T01:25:29.060 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-08-19T01:25:29.060 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000223113 s, 2.3 MB/s 2024-08-19T01:25:29.062 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T01:25:29.091 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:28 smithi161 podman[23336]: 2024-08-19 01:25:28.723188949 +0000 UTC m=+0.017097592 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:25:29.091 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:28 smithi161 podman[23336]: 2024-08-19 01:25:28.845845007 +0000 UTC m=+0.139753634 container create feace770023ab161a854ce28dcc016e0c0ce14129698c9c14bf4a8d67e06424b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-08-19T01:25:29.091 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 podman[23336]: 2024-08-19 01:25:29.090098229 +0000 UTC m=+0.384006859 container init feace770023ab161a854ce28dcc016e0c0ce14129698c9c14bf4a8d67e06424b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-08-19T01:25:29.107 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-08-19T01:25:29.162 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T01:25:29.162 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:29.162 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 712 Links: 1 2024-08-19T01:25:29.162 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:29.163 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:29.163 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-08-19 01:25:28.541061581 +0000 2024-08-19T01:25:29.163 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-08-19 01:20:52.379123145 +0000 2024-08-19T01:25:29.163 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-08-19 01:20:52.379123145 +0000 2024-08-19T01:25:29.163 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-08-19 01:20:52.379123145 +0000 2024-08-19T01:25:29.163 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T01:25:29.228 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-08-19T01:25:29.228 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-08-19T01:25:29.228 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000212621 s, 2.4 MB/s 2024-08-19T01:25:29.230 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T01:25:29.286 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 729 Links: 1 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-08-19 01:25:28.541061581 +0000 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-08-19 01:20:52.705117322 +0000 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-08-19 01:20:52.705117322 +0000 2024-08-19T01:25:29.341 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-08-19 01:20:52.705117322 +0000 2024-08-19T01:25:29.342 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T01:25:29.394 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 podman[23336]: 2024-08-19 01:25:29.093168804 +0000 UTC m=+0.387077432 container start feace770023ab161a854ce28dcc016e0c0ce14129698c9c14bf4a8d67e06424b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-08-19T01:25:29.394 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mgr[23459]: -- 172.21.15.161:0/1990937201 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3f265f0e0 con 0x55f3f263c800 2024-08-19T01:25:29.394 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 bash[23336]: feace770023ab161a854ce28dcc016e0c0ce14129698c9c14bf4a8d67e06424b 2024-08-19T01:25:29.394 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 systemd[1]: Started Ceph mgr.z for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:25:29.394 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:29.322+0000 7f9741b15040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:25:29.394 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:29.391+0000 7f9741b15040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:25:29.405 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-08-19T01:25:29.405 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-08-19T01:25:29.405 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000207399 s, 2.5 MB/s 2024-08-19T01:25:29.406 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T01:25:29.463 INFO:tasks.cephadm:Deploying osd.0 on smithi064 with /dev/vg_nvme/lv_4... 2024-08-19T01:25:29.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-19T01:25:29.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:29.540+0000 7f9741b15040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:25:29.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:29.697+0000 7f9741b15040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:25:29.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:29.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:29.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T01:25:29.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:29.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:29.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:29.949 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:29.839+0000 7f9741b15040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:25:30.443 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:29.963+0000 7f9741b15040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:25:30.443 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.087+0000 7f9741b15040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:25:30.443 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.143+0000 7f9741b15040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:25:30.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-mon[21757]: Deploying daemon mgr.y on smithi123 2024-08-19T01:25:30.700 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.441+0000 7f9741b15040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:25:30.700 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.505+0000 7f9741b15040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:25:30.700 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.572+0000 7f9741b15040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:25:30.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:30 smithi064 ceph-mon[18515]: Deploying daemon mgr.y on smithi123 2024-08-19T01:25:30.721 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:30 smithi123 ceph-mon[21829]: Deploying daemon mgr.y on smithi123 2024-08-19T01:25:31.032 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.720+0000 7f9741b15040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:25:31.032 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.779+0000 7f9741b15040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:25:31.032 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:30.844+0000 7f9741b15040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:25:31.032 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:31.030+0000 7f9741b15040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:25:31.136 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:30 smithi123 systemd[1]: Starting Ceph mgr.y for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:25:31.399 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 podman[23184]: 2024-08-19 01:25:31.135046293 +0000 UTC m=+0.110100786 container create 6c99f9ff5c165213516e6f9df5703104f9eeb3aa4fde080bb8185d389f7f53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-08-19T01:25:31.399 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 podman[23184]: 2024-08-19 01:25:31.041901293 +0000 UTC m=+0.016955788 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:25:31.399 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 podman[23184]: 2024-08-19 01:25:31.370717062 +0000 UTC m=+0.345771562 container init 6c99f9ff5c165213516e6f9df5703104f9eeb3aa4fde080bb8185d389f7f53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-19T01:25:31.399 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 podman[23184]: 2024-08-19 01:25:31.373488007 +0000 UTC m=+0.348542500 container start 6c99f9ff5c165213516e6f9df5703104f9eeb3aa4fde080bb8185d389f7f53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-08-19T01:25:31.427 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:31.090+0000 7f9741b15040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:25:31.427 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:31.150+0000 7f9741b15040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:25:31.427 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:31.217+0000 7f9741b15040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:25:31.428 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:31.424+0000 7f9741b15040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:25:31.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:31 smithi123 ceph-mon[21829]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:31.659 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 ceph-mgr[23203]: -- 172.21.15.123:0/2318441491 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b927bd0e0 con 0x556b9279b000 2024-08-19T01:25:31.659 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 bash[23184]: 6c99f9ff5c165213516e6f9df5703104f9eeb3aa4fde080bb8185d389f7f53cb 2024-08-19T01:25:31.659 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 systemd[1]: Started Ceph mgr.y for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:25:31.659 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:31.588+0000 7f53241ce040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:25:31.659 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:31.656+0000 7f53241ce040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:25:31.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:31 smithi161 ceph-mon[21757]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:31.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:31.485+0000 7f9741b15040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:25:31.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:31 smithi064 ceph-mon[18515]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:31.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:31.805+0000 7f53241ce040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:25:31.991 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:25:32.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2024-08-19T01:25:32.246 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:31 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:31.973+0000 7f53241ce040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:25:32.246 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:32.125+0000 7f53241ce040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:25:32.263 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:25:32.378 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:32.056+0000 7f9741b15040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:25:32.540 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:32.252+0000 7f53241ce040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:25:32.540 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:32.385+0000 7f53241ce040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:25:32.540 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:32.443+0000 7f53241ce040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:25:32.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:32.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.680+0000 7f1d81169640 1 -- 172.21.15.64:0/758068994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 msgr2=0x7f1d7c101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.680+0000 7f1d81169640 1 --2- 172.21.15.64:0/758068994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 0x7f1d7c101eb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009900 tx=0x7f1d7002a410 comp rx=0 tx=0).stop 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.680+0000 7f1d81169640 1 -- 172.21.15.64:0/758068994 shutdown_connections 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.680+0000 7f1d81169640 1 --2- 172.21.15.64:0/758068994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c1023f0 0x7f1d7c108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.680+0000 7f1d81169640 1 --2- 172.21.15.64:0/758068994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 0x7f1d7c101eb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.680+0000 7f1d81169640 1 --2- 172.21.15.64:0/758068994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d7c100830 0x7f1d7c100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.680+0000 7f1d81169640 1 -- 172.21.15.64:0/758068994 >> 172.21.15.64:0/758068994 conn(0x7f1d7c0fbfc0 msgr2=0x7f1d7c0fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.681+0000 7f1d81169640 1 -- 172.21.15.64:0/758068994 shutdown_connections 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.681+0000 7f1d81169640 1 -- 172.21.15.64:0/758068994 wait complete. 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.681+0000 7f1d81169640 1 Processor -- start 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d81169640 1 -- start start 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d81169640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c100830 0x7f1d7c06b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:32.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d81169640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 0x7f1d7c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d81169640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d7c1023f0 0x7f1d7c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d81169640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c10c4b0 con 0x7f1d7c101a30 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d81169640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c10c330 con 0x7f1d7c1023f0 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d81169640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c10c630 con 0x7f1d7c100830 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 0x7f1d7c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c100830 0x7f1d7c06b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c100830 0x7f1d7c06b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:33580/0 (socket says 172.21.15.64:33580) 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 0x7f1d7c19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49000/0 (socket says 172.21.15.64:49000) 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 -- 172.21.15.64:0/3558352090 learned_addr learned my addr 172.21.15.64:0/3558352090 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7b59e640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d7c1023f0 0x7f1d7c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 -- 172.21.15.64:0/3558352090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d7c1023f0 msgr2=0x7f1d7c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d7c1023f0 0x7f1d7c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 -- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 msgr2=0x7f1d7c19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:32.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 0x7f1d7c19c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 -- 172.21.15.64:0/3558352090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c19f8d0 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.682+0000 7f1d7ad9d640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c100830 0x7f1d7c06b020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00e9c0 tx=0x7f1d6c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.683+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00cd80 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.683+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7c19fb60 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.683+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d6c0035f0 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.683+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c010690 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.683+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7c1a00c0 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.684+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1d6c010830 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.684+0000 7f1d63fff640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1d64039290 0x7f1d6403b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.684+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1d6c04d8b0 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.684+0000 7f1d7a59c640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1d64039290 0x7f1d6403b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.684+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d7c107c80 con 0x7f1d7c100830 2024-08-19T01:25:32.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.684+0000 7f1d7a59c640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1d64039290 0x7f1d6403b750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1d700314e0 tx=0x7f1d70000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:32.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.693+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1d6c0173d0 con 0x7f1d7c100830 2024-08-19T01:25:32.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:32.378+0000 7f9741b15040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:25:32.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:32.828 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:32.762+0000 7f53241ce040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:25:32.828 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:32.825+0000 7f53241ce040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:25:32.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:32.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:32.882+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1d7c062360 con 0x7f1d64039290 2024-08-19T01:25:33.098 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:32.893+0000 7f53241ce040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:25:33.098 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.038+0000 7f53241ce040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:25:33.099 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.096+0000 7f53241ce040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:25:33.199 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:32.820+0000 7f9741b15040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:25:33.199 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:32.880+0000 7f9741b15040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:25:33.200 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:25:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:25:32.939+0000 7f9741b15040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:25:33.406 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.161+0000 7f53241ce040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:25:33.407 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.344+0000 7f53241ce040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:25:33.407 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.404+0000 7f53241ce040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:25:33.666 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.463+0000 7f53241ce040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:25:33.666 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.528+0000 7f53241ce040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:25:33.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:25:33.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:33.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:25:33.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:25:33.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:25:33.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:25:33.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:34.166 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.740+0000 7f53241ce040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:25:34.166 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:33.799+0000 7f53241ce040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:25:34.626 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:34.316+0000 7f53241ce040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:25:34.626 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:34.623+0000 7f53241ce040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:25:34.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: Reconfiguring mgr.x (unknown last config time)... 2024-08-19T01:25:34.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: Reconfiguring daemon mgr.x on smithi064 2024-08-19T01:25:34.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:34.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:34.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: Standby manager daemon z started 2024-08-19T01:25:34.627 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-08-19T01:25:34.627 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:25:34.627 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-08-19T01:25:34.627 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:34 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:25:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:34.641+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f1d6c018ce0 con 0x7f1d7c100830 2024-08-19T01:25:34.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: Reconfiguring mgr.x (unknown last config time)... 2024-08-19T01:25:34.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: Reconfiguring daemon mgr.x on smithi064 2024-08-19T01:25:34.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:34.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:34.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: Standby manager daemon z started 2024-08-19T01:25:34.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-08-19T01:25:34.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:25:34.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-08-19T01:25:34.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:34 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: Reconfiguring mgr.x (unknown last config time)... 2024-08-19T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: Reconfiguring daemon mgr.x on smithi064 2024-08-19T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: Standby manager daemon z started 2024-08-19T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-08-19T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-08-19T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:34 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1739303439' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:25:35.416 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:35.040+0000 7f53241ce040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:25:35.416 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:35.095+0000 7f53241ce040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:25:35.416 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:25:35.154+0000 7f53241ce040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: mgrmap e14: x(active, since 61s), standbys: z 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:35.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:35.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:35.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: mgrmap e14: x(active, since 61s), standbys: z 2024-08-19T01:25:35.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-08-19T01:25:35.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:35.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:35.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:35.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: mgrmap e14: x(active, since 61s), standbys: z 2024-08-19T01:25:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-08-19T01:25:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4067524162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c3fb9c1-436e-47fd-ae54-4f88a7b757c0"}]: dispatch 2024-08-19T01:25:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4067524162' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c3fb9c1-436e-47fd-ae54-4f88a7b757c0"}]': finished 2024-08-19T01:25:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: osdmap e5: 1 total, 0 up, 1 in 2024-08-19T01:25:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: Standby manager daemon y started 2024-08-19T01:25:36.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-19T01:25:36.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:25:36.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-19T01:25:36.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:36 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:25:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4067524162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c3fb9c1-436e-47fd-ae54-4f88a7b757c0"}]: dispatch 2024-08-19T01:25:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4067524162' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c3fb9c1-436e-47fd-ae54-4f88a7b757c0"}]': finished 2024-08-19T01:25:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: osdmap e5: 1 total, 0 up, 1 in 2024-08-19T01:25:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: Standby manager daemon y started 2024-08-19T01:25:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-19T01:25:36.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:25:36.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-19T01:25:36.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:36 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:25:36.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4067524162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c3fb9c1-436e-47fd-ae54-4f88a7b757c0"}]: dispatch 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4067524162' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c3fb9c1-436e-47fd-ae54-4f88a7b757c0"}]': finished 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: osdmap e5: 1 total, 0 up, 1 in 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: Standby manager daemon y started 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-19T01:25:36.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:36 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/3190580215' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:25:37.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:37.201+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mon.1 v2:172.21.15.161:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f1d6c00f3c0 con 0x7f1d7c100830 2024-08-19T01:25:37.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:37 smithi123 ceph-mon[21829]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:37.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:37 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3813290093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:25:37.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:37 smithi123 ceph-mon[21829]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-08-19T01:25:37.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T01:25:37.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:37 smithi161 ceph-mon[21757]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:37.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:37 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3813290093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:25:37.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:37 smithi161 ceph-mon[21757]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-08-19T01:25:37.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T01:25:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:37 smithi064 ceph-mon[18515]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:37 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3813290093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:25:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:37 smithi064 ceph-mon[18515]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-08-19T01:25:37.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T01:25:39.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:39 smithi123 ceph-mon[21829]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:39.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:39 smithi161 ceph-mon[21757]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:39.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:39 smithi064 ceph-mon[18515]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:41.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:41 smithi064 ceph-mon[18515]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:41.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:41 smithi123 ceph-mon[21829]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:41.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:41 smithi161 ceph-mon[21757]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:43.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:43 smithi064 ceph-mon[18515]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:43.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-19T01:25:43.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:43.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:43 smithi123 ceph-mon[21829]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:43.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-19T01:25:43.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:43 smithi161 ceph-mon[21757]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-19T01:25:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:44.607 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:44 smithi064 ceph-mon[18515]: Deploying daemon osd.0 on smithi064 2024-08-19T01:25:44.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:44 smithi123 ceph-mon[21829]: Deploying daemon osd.0 on smithi064 2024-08-19T01:25:44.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:44 smithi161 ceph-mon[21757]: Deploying daemon osd.0 on smithi064 2024-08-19T01:25:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:45 smithi064 ceph-mon[18515]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:45.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:45 smithi123 ceph-mon[21829]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:45.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:45 smithi161 ceph-mon[21757]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:47.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:47 smithi064 ceph-mon[18515]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:47.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:47.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:47.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:47 smithi123 ceph-mon[21829]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:47.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:47 smithi161 ceph-mon[21757]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:47.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:47.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:47.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:49 smithi123 ceph-mon[21829]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:49.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:49 smithi161 ceph-mon[21757]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:49.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:49 smithi064 ceph-mon[18515]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:50.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.444+0000 7f1d63fff640 1 -- 172.21.15.64:0/3558352090 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1d7c062360 con 0x7f1d64039290 2024-08-19T01:25:50.447 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 0 on host 'smithi064' 2024-08-19T01:25:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1d64039290 msgr2=0x7f1d6403b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1d64039290 0x7f1d6403b750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1d700314e0 tx=0x7f1d70000bc0 comp rx=0 tx=0).stop 2024-08-19T01:25:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c100830 msgr2=0x7f1d7c06b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c100830 0x7f1d7c06b020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00e9c0 tx=0x7f1d6c00ee80 comp rx=0 tx=0).stop 2024-08-19T01:25:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 shutdown_connections 2024-08-19T01:25:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1d64039290 0x7f1d6403b750 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d7c1023f0 0x7f1d7c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d7c101a30 0x7f1d7c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 --2- 172.21.15.64:0/3558352090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d7c100830 0x7f1d7c06b020 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 >> 172.21.15.64:0/3558352090 conn(0x7f1d7c0fbfc0 msgr2=0x7f1d7c104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 shutdown_connections 2024-08-19T01:25:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:25:50.446+0000 7f1d81169640 1 -- 172.21.15.64:0/3558352090 wait complete. 2024-08-19T01:25:51.088 DEBUG:teuthology.orchestra.run.smithi064:osd.0> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.0.service 2024-08-19T01:25:51.090 INFO:tasks.cephadm:Deploying osd.1 on smithi123 with /dev/vg_nvme/lv_4... 2024-08-19T01:25:51.090 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-19T01:25:51.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:51 smithi064 ceph-mon[18515]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:51.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.349 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:25:51.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:51 smithi123 ceph-mon[21829]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:51.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:51 smithi161 ceph-mon[21757]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:51.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:51.606 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:25:51 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0[29576]: 2024-08-19T01:25:51.333+0000 7f74731a3640 -1 osd.0 0 log_to_monitors true 2024-08-19T01:25:52.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:52 smithi064 ceph-mon[18515]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-19T01:25:52.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:52 smithi123 ceph-mon[21829]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-19T01:25:52.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:52 smithi161 ceph-mon[21757]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-19T01:25:53.370 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:53 smithi161 ceph-mon[21757]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:53.370 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:53 smithi161 ceph-mon[21757]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-19T01:25:53.370 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:53 smithi161 ceph-mon[21757]: osdmap e6: 1 total, 0 up, 1 in 2024-08-19T01:25:53.370 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:53 smithi161 ceph-mon[21757]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:25:53.370 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:53.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:53 smithi123 ceph-mon[21829]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:53.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:53 smithi123 ceph-mon[21829]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-19T01:25:53.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:53 smithi123 ceph-mon[21829]: osdmap e6: 1 total, 0 up, 1 in 2024-08-19T01:25:53.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:53 smithi123 ceph-mon[21829]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:25:53.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:53.485 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:25:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0[29576]: 2024-08-19T01:25:53.114+0000 7f746f126640 -1 osd.0 0 waiting for initial osdmap 2024-08-19T01:25:53.485 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:25:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0[29576]: 2024-08-19T01:25:53.122+0000 7f746af4e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:25:53.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:53 smithi064 ceph-mon[18515]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:53.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:53 smithi064 ceph-mon[18515]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-19T01:25:53.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:53 smithi064 ceph-mon[18515]: osdmap e6: 1 total, 0 up, 1 in 2024-08-19T01:25:53.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:53 smithi064 ceph-mon[18515]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:25:53.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:53.509 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:25:53.534 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi123:vg_nvme/lv_4 2024-08-19T01:25:53.777 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:25:54.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:25:54.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: osdmap e7: 1 total, 0 up, 1 in 2024-08-19T01:25:54.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:54.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:25:54.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: osdmap e7: 1 total, 0 up, 1 in 2024-08-19T01:25:54.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:54.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:54.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:25:54.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:54.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:54.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 -- 172.21.15.123:0/3423925799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c100800 msgr2=0x7f425c100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 --2- 172.21.15.123:0/3423925799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c100800 0x7f425c100c00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4250009900 tx=0x7f425002a410 comp rx=0 tx=0).stop 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 -- 172.21.15.123:0/3423925799 shutdown_connections 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 --2- 172.21.15.123:0/3423925799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f425c1023c0 0x7f425c108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 --2- 172.21.15.123:0/3423925799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f425c101a00 0x7f425c101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 --2- 172.21.15.123:0/3423925799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c100800 0x7f425c100c00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 -- 172.21.15.123:0/3423925799 >> 172.21.15.123:0/3423925799 conn(0x7f425c0fbfb0 msgr2=0x7f425c0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 -- 172.21.15.123:0/3423925799 shutdown_connections 2024-08-19T01:25:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 -- 172.21.15.123:0/3423925799 wait complete. 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 Processor -- start 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.251+0000 7f4262d13640 1 -- start start 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4262d13640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f425c100800 0x7f425c06a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4262d13640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c101a00 0x7f425c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4262d13640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f425c1023c0 0x7f425c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4262d13640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f425c10c460 con 0x7f425c100800 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4262d13640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f425c10c2e0 con 0x7f425c101a00 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4262d13640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f425c10c5e0 con 0x7f425c1023c0 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c101a00 0x7f425c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c101a00 0x7f425c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33274/0 (socket says 172.21.15.123:33274) 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 -- 172.21.15.123:0/410602632 learned_addr learned my addr 172.21.15.123:0/410602632 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4260aaf640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f425c100800 0x7f425c06a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:54.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f42612b0640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f425c1023c0 0x7f425c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 -- 172.21.15.123:0/410602632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f425c1023c0 msgr2=0x7f425c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f425c1023c0 0x7f425c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 -- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f425c100800 msgr2=0x7f425c06a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f425c100800 0x7f425c06a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 -- 172.21.15.123:0/410602632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425c19f8b0 con 0x7f425c101a00 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f425bfff640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c101a00 0x7f425c19c980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f424800e9f0 tx=0x7f424800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4259ffb640 1 -- 172.21.15.123:0/410602632 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424800cd80 con 0x7f425c101a00 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4259ffb640 1 -- 172.21.15.123:0/410602632 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4248010910 con 0x7f425c101a00 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425c19fb40 con 0x7f425c101a00 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.252+0000 7f4259ffb640 1 -- 172.21.15.123:0/410602632 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4248018990 con 0x7f425c101a00 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.253+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f425c1a0080 con 0x7f425c101a00 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.254+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f425c107c50 con 0x7f425c101a00 2024-08-19T01:25:54.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.255+0000 7f4259ffb640 1 -- 172.21.15.123:0/410602632 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f4248010ac0 con 0x7f425c101a00 2024-08-19T01:25:54.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.256+0000 7f4259ffb640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f424c0a2af0 0x7f424c0a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:25:54.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.256+0000 7f4259ffb640 1 -- 172.21.15.123:0/410602632 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f42480cd890 con 0x7f425c101a00 2024-08-19T01:25:54.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.256+0000 7f4260aaf640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f424c0a2af0 0x7f424c0a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:25:54.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.257+0000 7f4260aaf640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f424c0a2af0 0x7f424c0a4fb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4250005ab0 tx=0x7f4250000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:25:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.261+0000 7f4259ffb640 1 -- 172.21.15.123:0/410602632 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f424809bc10 con 0x7f425c101a00 2024-08-19T01:25:54.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:25:54.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: osdmap e7: 1 total, 0 up, 1 in 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:54.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:25:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:25:54.449+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f425c062360 con 0x7f424c0a2af0 2024-08-19T01:25:55.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:25:55.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:25:55.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:55.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi064 2024-08-19T01:25:55.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi064 to 17088M 2024-08-19T01:25:55.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] boot 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: osdmap e8: 1 total, 1 up, 1 in 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:25:55.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:55.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi064 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi064 to 17088M 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] boot 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: osdmap e8: 1 total, 1 up, 1 in 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:55.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:25:55.451 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:25:55.451 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:55.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:25:55.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:25:55.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T01:25:55.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi064 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi064 to 17088M 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: osd.0 [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] boot 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: osdmap e8: 1 total, 1 up, 1 in 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:25:55.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:25:56.398 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:56 smithi123 ceph-mon[21829]: osdmap e9: 1 total, 1 up, 1 in 2024-08-19T01:25:56.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:56 smithi161 ceph-mon[21757]: osdmap e9: 1 total, 1 up, 1 in 2024-08-19T01:25:56.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:56 smithi064 ceph-mon[18515]: osdmap e9: 1 total, 1 up, 1 in 2024-08-19T01:25:57.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:57 smithi123 ceph-mon[21829]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:25:57.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:57 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/3823716289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]: dispatch 2024-08-19T01:25:57.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:57 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]: dispatch 2024-08-19T01:25:57.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:57 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]': finished 2024-08-19T01:25:57.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:57 smithi123 ceph-mon[21829]: osdmap e10: 2 total, 1 up, 2 in 2024-08-19T01:25:57.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:57 smithi161 ceph-mon[21757]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:57 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/3823716289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]: dispatch 2024-08-19T01:25:57.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:57 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]: dispatch 2024-08-19T01:25:57.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:57 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]': finished 2024-08-19T01:25:57.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:57 smithi161 ceph-mon[21757]: osdmap e10: 2 total, 1 up, 2 in 2024-08-19T01:25:57.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:25:57.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:57 smithi064 ceph-mon[18515]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:25:57.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:57 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/3823716289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]: dispatch 2024-08-19T01:25:57.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:57 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]: dispatch 2024-08-19T01:25:57.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:57 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e1f4596-0fe3-4857-a061-1b173ea09ea3"}]': finished 2024-08-19T01:25:57.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:57 smithi064 ceph-mon[18515]: osdmap e10: 2 total, 1 up, 2 in 2024-08-19T01:25:57.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:25:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:58 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/307821666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:25:58.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:58 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/307821666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:25:58.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:58 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/307821666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:25:59.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:25:59 smithi064 ceph-mon[18515]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:25:59.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:25:59 smithi123 ceph-mon[21829]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:25:59.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:25:59 smithi161 ceph-mon[21757]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:01.403 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:01 smithi123 ceph-mon[21829]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:01.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:01 smithi161 ceph-mon[21757]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:01.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:01 smithi064 ceph-mon[18515]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:03.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:03 smithi123 ceph-mon[21829]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:03.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:03 smithi161 ceph-mon[21757]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:03.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:03 smithi064 ceph-mon[18515]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:04.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:04 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-19T01:26:04.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:04 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:04.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:04 smithi123 ceph-mon[21829]: Deploying daemon osd.1 on smithi123 2024-08-19T01:26:04.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:04 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-19T01:26:04.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:04 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:04.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:04 smithi161 ceph-mon[21757]: Deploying daemon osd.1 on smithi123 2024-08-19T01:26:04.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:04 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-19T01:26:04.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:04 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:04.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:04 smithi064 ceph-mon[18515]: Deploying daemon osd.1 on smithi123 2024-08-19T01:26:05.356 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:05 smithi123 ceph-mon[21829]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:05.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:05 smithi161 ceph-mon[21757]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:05.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:05 smithi064 ceph-mon[18515]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:07 smithi123 ceph-mon[21829]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:07.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:07 smithi161 ceph-mon[21757]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:07.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:07 smithi064 ceph-mon[18515]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:09.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:09 smithi123 ceph-mon[21829]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:09.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:09.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:09.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:09.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:09 smithi161 ceph-mon[21757]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:09.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:09.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:09.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:09.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:09 smithi064 ceph-mon[18515]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:09.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:09.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:09.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:11.297 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:11 smithi123 ceph-mon[21829]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:11.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.335+0000 7f4259ffb640 1 -- 172.21.15.123:0/410602632 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f425c062360 con 0x7f424c0a2af0 2024-08-19T01:26:11.338 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 1 on host 'smithi123' 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f424c0a2af0 msgr2=0x7f424c0a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f424c0a2af0 0x7f424c0a4fb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4250005ab0 tx=0x7f4250000bc0 comp rx=0 tx=0).stop 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c101a00 msgr2=0x7f425c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c101a00 0x7f425c19c980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f424800e9f0 tx=0x7f424800eeb0 comp rx=0 tx=0).stop 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 shutdown_connections 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f424c0a2af0 0x7f424c0a4fb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f425c1023c0 0x7f425c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:11.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425c101a00 0x7f425c19c980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:11.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 --2- 172.21.15.123:0/410602632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f425c100800 0x7f425c06a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:11.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 >> 172.21.15.123:0/410602632 conn(0x7f425c0fbfb0 msgr2=0x7f425c104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:26:11.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 shutdown_connections 2024-08-19T01:26:11.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:11.338+0000 7f4262d13640 1 -- 172.21.15.123:0/410602632 wait complete. 2024-08-19T01:26:11.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:11 smithi161 ceph-mon[21757]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:11.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:11 smithi064 ceph-mon[18515]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:11.960 DEBUG:teuthology.orchestra.run.smithi123:osd.1> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.1.service 2024-08-19T01:26:11.962 INFO:tasks.cephadm:Deploying osd.2 on smithi161 with /dev/vg_nvme/lv_4... 2024-08-19T01:26:11.962 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-19T01:26:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:26:12.283 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1[26267]: 2024-08-19T01:26:12.117+0000 7f267dfa0640 -1 osd.1 0 log_to_monitors true 2024-08-19T01:26:12.594 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:12 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.594 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:12 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.594 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:12 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.594 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:12 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.594 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:12 smithi161 ceph-mon[21757]: from='osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T01:26:12.594 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:12 smithi161 ceph-mon[21757]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T01:26:12.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:12 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:12 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:12 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:12 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:12 smithi123 ceph-mon[21829]: from='osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T01:26:12.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:12 smithi123 ceph-mon[21829]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T01:26:12.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:12 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:12 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:12 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:12 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:12.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:12 smithi064 ceph-mon[18515]: from='osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T01:26:12.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:12 smithi064 ceph-mon[18515]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T01:26:13.592 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:13 smithi161 ceph-mon[21757]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:13.592 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:13 smithi161 ceph-mon[21757]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-19T01:26:13.592 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:13 smithi161 ceph-mon[21757]: osdmap e11: 2 total, 1 up, 2 in 2024-08-19T01:26:13.592 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:13.592 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:13 smithi161 ceph-mon[21757]: from='osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:26:13.592 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:13 smithi161 ceph-mon[21757]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:26:13.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-mon[21829]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:13.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-mon[21829]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-19T01:26:13.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-mon[21829]: osdmap e11: 2 total, 1 up, 2 in 2024-08-19T01:26:13.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:13.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-mon[21829]: from='osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:26:13.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-mon[21829]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:26:13.667 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1[26267]: 2024-08-19T01:26:13.411+0000 7f2679f23640 -1 osd.1 0 waiting for initial osdmap 2024-08-19T01:26:13.667 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1[26267]: 2024-08-19T01:26:13.416+0000 7f2675d4b640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:26:13.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:13 smithi064 ceph-mon[18515]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:13.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:13 smithi064 ceph-mon[18515]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-19T01:26:13.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:13 smithi064 ceph-mon[18515]: osdmap e11: 2 total, 1 up, 2 in 2024-08-19T01:26:13.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:13.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:13 smithi064 ceph-mon[18515]: from='osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:26:13.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:13 smithi064 ceph-mon[18515]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:26:14.465 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:14 smithi123 ceph-mon[21829]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:26:14.465 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:14 smithi123 ceph-mon[21829]: osdmap e12: 2 total, 1 up, 2 in 2024-08-19T01:26:14.466 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:14 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:14.466 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:14 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:14.611 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:14 smithi161 ceph-mon[21757]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:26:14.611 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:14 smithi161 ceph-mon[21757]: osdmap e12: 2 total, 1 up, 2 in 2024-08-19T01:26:14.612 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:14 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:14.612 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:14 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:14.615 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:26:14.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_4 2024-08-19T01:26:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:14 smithi064 ceph-mon[18515]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:26:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:14 smithi064 ceph-mon[18515]: osdmap e12: 2 total, 1 up, 2 in 2024-08-19T01:26:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:14 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:14 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:14.894 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:26:15.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 -- 172.21.15.161:0/4217817183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394103910 msgr2=0x7fb3940ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:15.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 --2- 172.21.15.161:0/4217817183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394103910 0x7fb3940ff9c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb388009900 tx=0x7fb38802a410 comp rx=0 tx=0).stop 2024-08-19T01:26:15.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 -- 172.21.15.161:0/4217817183 shutdown_connections 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 --2- 172.21.15.161:0/4217817183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb394100080 0x7fb394109f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 --2- 172.21.15.161:0/4217817183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394103910 0x7fb3940ff9c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 --2- 172.21.15.161:0/4217817183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb394102f40 0x7fb394103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 -- 172.21.15.161:0/4217817183 >> 172.21.15.161:0/4217817183 conn(0x7fb3940fb770 msgr2=0x7fb3940fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 -- 172.21.15.161:0/4217817183 shutdown_connections 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.346+0000 7fb3991fd640 1 -- 172.21.15.161:0/4217817183 wait complete. 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.347+0000 7fb3991fd640 1 Processor -- start 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.347+0000 7fb3991fd640 1 -- start start 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb3991fd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb394100080 0x7fb394197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb3991fd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394102f40 0x7fb3941981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb3991fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb394103910 0x7fb39419f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:15.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb3991fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb39410d5d0 con 0x7fb394103910 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb3991fd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb39410d450 con 0x7fb394100080 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb3991fd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb39410d750 con 0x7fb394102f40 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394102f40 0x7fb3941981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394102f40 0x7fb3941981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59616/0 (socket says 172.21.15.161:59616) 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 -- 172.21.15.161:0/3217018841 learned_addr learned my addr 172.21.15.161:0/3217018841 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb392d9d640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb394100080 0x7fb394197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39359e640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb394103910 0x7fb39419f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 -- 172.21.15.161:0/3217018841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb394100080 msgr2=0x7fb394197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb394100080 0x7fb394197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 -- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb394103910 msgr2=0x7fb39419f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb394103910 0x7fb39419f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:15.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 -- 172.21.15.161:0/3217018841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39419f880 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb392d9d640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb394100080 0x7fb394197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39359e640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb394103910 0x7fb39419f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb39259c640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394102f40 0x7fb3941981a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb388006010 tx=0x7fb3880042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb37bfff640 1 -- 172.21.15.161:0/3217018841 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38803f030 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb37bfff640 1 -- 172.21.15.161:0/3217018841 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb388003ce0 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39419fb10 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.348+0000 7fb37bfff640 1 -- 172.21.15.161:0/3217018841 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388035070 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.349+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39419fff0 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.350+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb394108f00 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.350+0000 7fb37bfff640 1 -- 172.21.15.161:0/3217018841 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fb388042030 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.351+0000 7fb37bfff640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb37c0a2af0 0x7fb37c0a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.351+0000 7fb37bfff640 1 -- 172.21.15.161:0/3217018841 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fb3880edca0 con 0x7fb394102f40 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.351+0000 7fb392d9d640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb37c0a2af0 0x7fb37c0a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:15.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.352+0000 7fb392d9d640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb37c0a2af0 0x7fb37c0a4fb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb394198b40 tx=0x7fb384005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:26:15.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.355+0000 7fb37bfff640 1 -- 172.21.15.161:0/3217018841 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb3880bc450 con 0x7fb394102f40 2024-08-19T01:26:15.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:15.531+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb394062360 con 0x7fb37c0a2af0 2024-08-19T01:26:15.533 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:26:15.533 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:26:15.533 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] boot 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: osdmap e13: 2 total, 2 up, 2 in 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:15.534 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:15.535 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:15.535 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:26:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:26:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] boot 2024-08-19T01:26:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: osdmap e13: 2 total, 2 up, 2 in 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:26:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:26:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T01:26:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: osd.1 [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] boot 2024-08-19T01:26:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: osdmap e13: 2 total, 2 up, 2 in 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:16.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:16 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi123 2024-08-19T01:26:16.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:16 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi123 to 17088M 2024-08-19T01:26:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:16 smithi161 ceph-mon[21757]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:16 smithi161 ceph-mon[21757]: osdmap e14: 2 total, 2 up, 2 in 2024-08-19T01:26:16.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:16 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi123 2024-08-19T01:26:16.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:16 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi123 to 17088M 2024-08-19T01:26:16.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:16 smithi064 ceph-mon[18515]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:16.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:16.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:16.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:16.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:16 smithi064 ceph-mon[18515]: osdmap e14: 2 total, 2 up, 2 in 2024-08-19T01:26:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:16 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi123 2024-08-19T01:26:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:16 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi123 to 17088M 2024-08-19T01:26:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:16 smithi123 ceph-mon[21829]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:16 smithi123 ceph-mon[21829]: osdmap e14: 2 total, 2 up, 2 in 2024-08-19T01:26:17.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:17 smithi161 ceph-mon[21757]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:17.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:17 smithi064 ceph-mon[18515]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:17 smithi123 ceph-mon[21829]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:18.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:18 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/3405671129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]: dispatch 2024-08-19T01:26:18.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:18 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]: dispatch 2024-08-19T01:26:18.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:18 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]': finished 2024-08-19T01:26:18.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:18 smithi161 ceph-mon[21757]: osdmap e15: 3 total, 2 up, 3 in 2024-08-19T01:26:18.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:18 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:18.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:18 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/1500798778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:26:18.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:18 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/3405671129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]: dispatch 2024-08-19T01:26:18.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:18 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]: dispatch 2024-08-19T01:26:18.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:18 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]': finished 2024-08-19T01:26:18.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:18 smithi064 ceph-mon[18515]: osdmap e15: 3 total, 2 up, 3 in 2024-08-19T01:26:18.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:18 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:18.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:18 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/1500798778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:26:18.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:18 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/3405671129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]: dispatch 2024-08-19T01:26:18.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:18 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]: dispatch 2024-08-19T01:26:18.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:18 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b86cd6b3-a20f-4b12-82ca-c926be9cf348"}]': finished 2024-08-19T01:26:18.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:18 smithi123 ceph-mon[21829]: osdmap e15: 3 total, 2 up, 3 in 2024-08-19T01:26:18.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:18 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:18.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:18 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/1500798778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:26:19.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:19 smithi161 ceph-mon[21757]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:19.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:19 smithi064 ceph-mon[18515]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:19 smithi123 ceph-mon[21829]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:21.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:21 smithi161 ceph-mon[21757]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:21.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:21 smithi064 ceph-mon[18515]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:21.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:21 smithi123 ceph-mon[21829]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:23.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:23 smithi161 ceph-mon[21757]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:23.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:23 smithi064 ceph-mon[18515]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:23.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:23 smithi123 ceph-mon[21829]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:24.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-19T01:26:24.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:24.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-19T01:26:24.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:24.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-19T01:26:24.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:25.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:25 smithi064 ceph-mon[18515]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:25.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:25 smithi064 ceph-mon[18515]: Deploying daemon osd.2 on smithi161 2024-08-19T01:26:25.736 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:25 smithi161 ceph-mon[21757]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:25.737 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:25 smithi161 ceph-mon[21757]: Deploying daemon osd.2 on smithi161 2024-08-19T01:26:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:25 smithi123 ceph-mon[21829]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:25 smithi123 ceph-mon[21829]: Deploying daemon osd.2 on smithi161 2024-08-19T01:26:27.706 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:27 smithi161 ceph-mon[21757]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:27 smithi064 ceph-mon[18515]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:27.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:27 smithi123 ceph-mon[21829]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:29 smithi064 ceph-mon[18515]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:29.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:29.773 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:29 smithi161 ceph-mon[21757]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:29.773 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:29.773 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:29.773 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:29.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:29 smithi123 ceph-mon[21829]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:29.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:29.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:29.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:31.546 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:31 smithi161 ceph-mon[21757]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:31.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:31 smithi064 ceph-mon[18515]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:31.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.843+0000 7fb37bfff640 1 -- 172.21.15.161:0/3217018841 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb394062360 con 0x7fb37c0a2af0 2024-08-19T01:26:31.845 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 2 on host 'smithi161' 2024-08-19T01:26:31.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb37c0a2af0 msgr2=0x7fb37c0a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:31.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb37c0a2af0 0x7fb37c0a4fb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb394198b40 tx=0x7fb384005f90 comp rx=0 tx=0).stop 2024-08-19T01:26:31.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394102f40 msgr2=0x7fb3941981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:31.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394102f40 0x7fb3941981a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb388006010 tx=0x7fb3880042f0 comp rx=0 tx=0).stop 2024-08-19T01:26:31.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 shutdown_connections 2024-08-19T01:26:31.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb37c0a2af0 0x7fb37c0a4fb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:31.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb394103910 0x7fb39419f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:31.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb394102f40 0x7fb3941981a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:31.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 --2- 172.21.15.161:0/3217018841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb394100080 0x7fb394197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:31.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 >> 172.21.15.161:0/3217018841 conn(0x7fb3940fb770 msgr2=0x7fb3941060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:26:31.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 shutdown_connections 2024-08-19T01:26:31.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:26:31.845+0000 7fb3991fd640 1 -- 172.21.15.161:0/3217018841 wait complete. 2024-08-19T01:26:31.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:31 smithi123 ceph-mon[21829]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:32.847 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:32 smithi161 ceph-mon[21757]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:32.847 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:32.847 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:32.874 DEBUG:teuthology.orchestra.run.smithi161:osd.2> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.2.service 2024-08-19T01:26:32.876 INFO:tasks.cephadm:Deploying osd.3 on smithi064 with /dev/vg_nvme/lv_3... 2024-08-19T01:26:32.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-19T01:26:33.145 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:26:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:32 smithi123 ceph-mon[21829]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:33.199 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2[26704]: 2024-08-19T01:26:33.048+0000 7fcca957e640 -1 osd.2 0 log_to_monitors true 2024-08-19T01:26:33.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:32 smithi064 ceph-mon[18515]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:33.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:33.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:33.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:33 smithi161 ceph-mon[21757]: from='osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T01:26:33.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:33 smithi161 ceph-mon[21757]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T01:26:34.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:34.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:34.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:33 smithi123 ceph-mon[21829]: from='osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T01:26:34.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:33 smithi123 ceph-mon[21829]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T01:26:34.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:34.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:34.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:33 smithi064 ceph-mon[18515]: from='osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T01:26:34.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:33 smithi064 ceph-mon[18515]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T01:26:35.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:34 smithi161 ceph-mon[21757]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:35.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:34 smithi161 ceph-mon[21757]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T01:26:35.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:34 smithi161 ceph-mon[21757]: from='osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:26:35.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:34 smithi161 ceph-mon[21757]: osdmap e16: 3 total, 2 up, 3 in 2024-08-19T01:26:35.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:34 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:35.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:34 smithi161 ceph-mon[21757]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:26:35.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:34 smithi064 ceph-mon[18515]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:35.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:34 smithi064 ceph-mon[18515]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T01:26:35.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:34 smithi064 ceph-mon[18515]: from='osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:26:35.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:34 smithi064 ceph-mon[18515]: osdmap e16: 3 total, 2 up, 3 in 2024-08-19T01:26:35.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:34 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:35.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:34 smithi064 ceph-mon[18515]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:26:35.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:34 smithi123 ceph-mon[21829]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:35.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:34 smithi123 ceph-mon[21829]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T01:26:35.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:34 smithi123 ceph-mon[21829]: from='osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:26:35.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:34 smithi123 ceph-mon[21829]: osdmap e16: 3 total, 2 up, 3 in 2024-08-19T01:26:35.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:34 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:35.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:34 smithi123 ceph-mon[21829]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:26:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:26:35.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2024-08-19T01:26:35.692 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:35 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2[26704]: 2024-08-19T01:26:35.365+0000 7fcca5501640 -1 osd.2 0 waiting for initial osdmap 2024-08-19T01:26:35.692 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:35 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2[26704]: 2024-08-19T01:26:35.369+0000 7fcc9d321640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:26:35.881 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:26:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:35 smithi161 ceph-mon[21757]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:26:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:35 smithi161 ceph-mon[21757]: osdmap e17: 3 total, 2 up, 3 in 2024-08-19T01:26:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:35.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:35 smithi161 ceph-mon[21757]: from='osd.2 ' entity='osd.2' 2024-08-19T01:26:36.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:35 smithi064 ceph-mon[18515]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:26:36.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:35 smithi064 ceph-mon[18515]: osdmap e17: 3 total, 2 up, 3 in 2024-08-19T01:26:36.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:36.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:36.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:35 smithi064 ceph-mon[18515]: from='osd.2 ' entity='osd.2' 2024-08-19T01:26:36.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 -- 172.21.15.64:0/2653345197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9718101b80 msgr2=0x7f9718108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:36.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2653345197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9718101b80 0x7f9718108410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9710009ca0 tx=0x7f971002a8c0 comp rx=0 tx=0).stop 2024-08-19T01:26:36.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 -- 172.21.15.64:0/2653345197 shutdown_connections 2024-08-19T01:26:36.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2653345197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9718101b80 0x7f9718108410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:36.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2653345197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97181011c0 0x7f9718101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:36.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2653345197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97180fffc0 0x7f97181003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:36.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 -- 172.21.15.64:0/2653345197 >> 172.21.15.64:0/2653345197 conn(0x7f97180fb770 msgr2=0x7f97180fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:26:36.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 -- 172.21.15.64:0/2653345197 shutdown_connections 2024-08-19T01:26:36.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.358+0000 7f971eb4c640 1 -- 172.21.15.64:0/2653345197 wait complete. 2024-08-19T01:26:36.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 Processor -- start 2024-08-19T01:26:36.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 -- start start 2024-08-19T01:26:36.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97180fffc0 0x7f9718103f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97181011c0 0x7f97181025a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9718101b80 0x7f9718102ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f971810b3a0 con 0x7f97181011c0 2024-08-19T01:26:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f971810b220 con 0x7f9718101b80 2024-08-19T01:26:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971eb4c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f971810b520 con 0x7f97180fffc0 2024-08-19T01:26:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f9717fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97181011c0 0x7f97181025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:36.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f9717fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97181011c0 0x7f97181025a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57800/0 (socket says 172.21.15.64:57800) 2024-08-19T01:26:36.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f9717fff640 1 -- 172.21.15.64:0/2495741867 learned_addr learned my addr 172.21.15.64:0/2495741867 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:26:36.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971d0e9640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9718101b80 0x7f9718102ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:36.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.359+0000 7f971c8e8640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97180fffc0 0x7f9718103f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:36.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9717fff640 1 -- 172.21.15.64:0/2495741867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97180fffc0 msgr2=0x7f9718103f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:36.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9717fff640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97180fffc0 0x7f9718103f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:36.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9717fff640 1 -- 172.21.15.64:0/2495741867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9718101b80 msgr2=0x7f9718102ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:36.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9717fff640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9718101b80 0x7f9718102ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:36.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9717fff640 1 -- 172.21.15.64:0/2495741867 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97181033a0 con 0x7f97181011c0 2024-08-19T01:26:36.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9717fff640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97181011c0 0x7f97181025a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f970400c890 tx=0x7f970400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:26:36.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704004220 con 0x7f97181011c0 2024-08-19T01:26:36.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97040043c0 con 0x7f97181011c0 2024-08-19T01:26:36.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97181a1df0 con 0x7f97181011c0 2024-08-19T01:26:36.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97040049e0 con 0x7f97181011c0 2024-08-19T01:26:36.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.360+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97181a22a0 con 0x7f97181011c0 2024-08-19T01:26:36.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.361+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97181a2530 con 0x7f97181011c0 2024-08-19T01:26:36.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.362+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f97040079b0 con 0x7f97181011c0 2024-08-19T01:26:36.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.363+0000 7f9715ffb640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f97080a2b70 0x7f97080a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:36.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.363+0000 7f971c8e8640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f97080a2b70 0x7f97080a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:36.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.363+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2571+0+0 (secure 0 0 0) 0x7f97040cd950 con 0x7f97181011c0 2024-08-19T01:26:36.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.363+0000 7f971c8e8640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f97080a2b70 0x7f97080a5030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f970c005fd0 tx=0x7f970c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:26:36.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.367+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f9704014030 con 0x7f97181011c0 2024-08-19T01:26:36.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:35 smithi123 ceph-mon[21829]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:26:36.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:35 smithi123 ceph-mon[21829]: osdmap e17: 3 total, 2 up, 3 in 2024-08-19T01:26:36.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:36.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:36.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:35 smithi123 ceph-mon[21829]: from='osd.2 ' entity='osd.2' 2024-08-19T01:26:36.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:36.552+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f97181a2700 con 0x7f97080a2b70 2024-08-19T01:26:37.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi161 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi161 to 17088M 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] boot 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: osdmap e18: 3 total, 3 up, 3 in 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:37.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:26:37.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:26:37.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:37.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:37.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi161 2024-08-19T01:26:37.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi161 to 17088M 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] boot 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: osdmap e18: 3 total, 3 up, 3 in 2024-08-19T01:26:37.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:37.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:37.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:37.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:37.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:26:37.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:26:37.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi161 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi161 to 17088M 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:37.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: osd.2 [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] boot 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: osdmap e18: 3 total, 3 up, 3 in 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:37.418 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:38.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:26:38.201 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:26:38.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:26:39.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:38 smithi161 ceph-mon[21757]: pgmap v70: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:39.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-19T01:26:39.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:38 smithi161 ceph-mon[21757]: osdmap e19: 3 total, 3 up, 3 in 2024-08-19T01:26:39.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:26:39.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:38 smithi064 ceph-mon[18515]: pgmap v70: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:39.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-19T01:26:39.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:38 smithi064 ceph-mon[18515]: osdmap e19: 3 total, 3 up, 3 in 2024-08-19T01:26:39.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:26:39.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:38 smithi123 ceph-mon[21829]: pgmap v70: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:39.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-19T01:26:39.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:38 smithi123 ceph-mon[21829]: osdmap e19: 3 total, 3 up, 3 in 2024-08-19T01:26:39.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:26:39.567 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34444]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T01:26:39.568 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34444]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:39.568 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34444]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:39.568 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34444]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:39.940 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34466]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T01:26:39.940 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34466]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:39.940 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34466]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:39.940 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:26:39 smithi064 sudo[34466]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:40.090 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:39 smithi123 sudo[30101]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T01:26:40.090 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:39 smithi123 sudo[30101]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:40.090 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:39 smithi123 sudo[30101]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:40.090 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:39 smithi123 sudo[30101]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:40.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-19T01:26:40.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: osdmap e20: 3 total, 3 up, 3 in 2024-08-19T01:26:40.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1831766948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9312a92-2ed5-4111-8e98-1822e682fe2c"}]: dispatch 2024-08-19T01:26:40.091 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1831766948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9312a92-2ed5-4111-8e98-1822e682fe2c"}]': finished 2024-08-19T01:26:40.091 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: osdmap e21: 4 total, 3 up, 4 in 2024-08-19T01:26:40.091 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:40.091 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:40.091 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:39 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1850059131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:26:40.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-19T01:26:40.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: osdmap e20: 3 total, 3 up, 3 in 2024-08-19T01:26:40.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1831766948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9312a92-2ed5-4111-8e98-1822e682fe2c"}]: dispatch 2024-08-19T01:26:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1831766948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9312a92-2ed5-4111-8e98-1822e682fe2c"}]': finished 2024-08-19T01:26:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: osdmap e21: 4 total, 3 up, 4 in 2024-08-19T01:26:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:39 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1850059131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:26:40.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-19T01:26:40.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: osdmap e20: 3 total, 3 up, 3 in 2024-08-19T01:26:40.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1831766948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9312a92-2ed5-4111-8e98-1822e682fe2c"}]: dispatch 2024-08-19T01:26:40.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1831766948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9312a92-2ed5-4111-8e98-1822e682fe2c"}]': finished 2024-08-19T01:26:40.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: osdmap e21: 4 total, 3 up, 4 in 2024-08-19T01:26:40.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:40.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:40.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:39 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1850059131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:26:40.416 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:40 smithi123 sudo[30104]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T01:26:40.416 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:40 smithi123 sudo[30104]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:40.416 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:40 smithi123 sudo[30104]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:40.416 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:26:40 smithi123 sudo[30104]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:40.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:40.602+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9704097f70 con 0x7f97181011c0 2024-08-19T01:26:40.618 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30464]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T01:26:40.619 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30464]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:40.619 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30464]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:40.619 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30464]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:40.949 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30467]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T01:26:40.949 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:40.950 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30467]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:40.950 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:26:40 smithi161 sudo[30467]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:41.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:40 smithi064 sudo[34884]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T01:26:41.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:40 smithi064 sudo[34884]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:41.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:40 smithi064 sudo[34884]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:41.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:40 smithi064 sudo[34884]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:41.438 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 ceph-mon[21757]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:41.438 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:41.438 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 ceph-mon[21757]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-08-19T01:26:41.438 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:41.439 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 ceph-mon[21757]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:41.439 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 ceph-mon[21757]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:41.439 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T01:26:41.439 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:41.439 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30470]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:41.439 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30470]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 sudo[35070]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 sudo[35070]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 sudo[35070]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 sudo[35070]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 ceph-mon[18515]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 ceph-mon[18515]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:41.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 ceph-mon[18515]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:41.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:41 smithi064 ceph-mon[18515]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 ceph-mon[21829]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 ceph-mon[21829]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-08-19T01:26:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:41.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 ceph-mon[21829]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:41.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 ceph-mon[21829]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:41.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30473]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T01:26:41.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:41.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30473]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:41.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:41 smithi161 sudo[30473]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:42.176 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 sudo[30107]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T01:26:42.176 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 sudo[30107]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:42.176 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 sudo[30107]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:42.176 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:41 smithi123 sudo[30107]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:42.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.451 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.451 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.451 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:42 smithi161 ceph-mon[21757]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:42.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:42.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:42 smithi064 ceph-mon[18515]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:42.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 sudo[30110]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T01:26:42.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 sudo[30110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T01:26:42.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 sudo[30110]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 sudo[30110]: pam_unix(sudo:session): session closed for user root 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:26:42.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:42 smithi123 ceph-mon[21829]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T01:26:43.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:43 smithi064 ceph-mon[18515]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:43.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:43 smithi161 ceph-mon[21757]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:43.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:43 smithi123 ceph-mon[21829]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:45.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:45 smithi161 ceph-mon[21757]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:45.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:45 smithi064 ceph-mon[18515]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:45 smithi123 ceph-mon[21829]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:46.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:46 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-19T01:26:46.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:46 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:46.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:46 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-19T01:26:46.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:46 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:46 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-19T01:26:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:46 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:47.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:47 smithi161 ceph-mon[21757]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 196 KiB/s wr, 14 op/s 2024-08-19T01:26:47.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:47 smithi161 ceph-mon[21757]: Deploying daemon osd.3 on smithi064 2024-08-19T01:26:47.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:47 smithi064 ceph-mon[18515]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 196 KiB/s wr, 14 op/s 2024-08-19T01:26:47.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:47 smithi064 ceph-mon[18515]: Deploying daemon osd.3 on smithi064 2024-08-19T01:26:47.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:47 smithi123 ceph-mon[21829]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 196 KiB/s wr, 14 op/s 2024-08-19T01:26:47.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:47 smithi123 ceph-mon[21829]: Deploying daemon osd.3 on smithi064 2024-08-19T01:26:49.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:49 smithi064 ceph-mon[18515]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-08-19T01:26:49.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:49 smithi123 ceph-mon[21829]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-08-19T01:26:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:49 smithi161 ceph-mon[21757]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-08-19T01:26:51.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:51 smithi064 ceph-mon[18515]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2024-08-19T01:26:51.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:51.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:51.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:51.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:51 smithi123 ceph-mon[21829]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2024-08-19T01:26:51.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:51.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:51.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:51.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:51 smithi161 ceph-mon[21757]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2024-08-19T01:26:51.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:51.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:51.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:53.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:53 smithi064 ceph-mon[18515]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-08-19T01:26:53.501 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:53 smithi161 ceph-mon[21757]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-08-19T01:26:53.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:53 smithi123 ceph-mon[21829]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-08-19T01:26:54.229 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 3 on host 'smithi064' 2024-08-19T01:26:54.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.227+0000 7f9715ffb640 1 -- 172.21.15.64:0/2495741867 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f97181a2700 con 0x7f97080a2b70 2024-08-19T01:26:54.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.229+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f97080a2b70 msgr2=0x7f97080a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.229+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f97080a2b70 0x7f97080a5030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f970c005fd0 tx=0x7f970c005850 comp rx=0 tx=0).stop 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.229+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97181011c0 msgr2=0x7f97181025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.229+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97181011c0 0x7f97181025a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f970400c890 tx=0x7f970400cd50 comp rx=0 tx=0).stop 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 shutdown_connections 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f97080a2b70 0x7f97080a5030 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9718101b80 0x7f9718102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97181011c0 0x7f97181025a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 --2- 172.21.15.64:0/2495741867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97180fffc0 0x7f9718103f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 >> 172.21.15.64:0/2495741867 conn(0x7f97180fb770 msgr2=0x7f97181066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 shutdown_connections 2024-08-19T01:26:54.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:26:54.230+0000 7f971eb4c640 1 -- 172.21.15.64:0/2495741867 wait complete. 2024-08-19T01:26:54.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:54.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:54.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:54.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:54.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:54.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:54.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.864 DEBUG:teuthology.orchestra.run.smithi064:osd.3> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.3.service 2024-08-19T01:26:54.866 INFO:tasks.cephadm:Deploying osd.4 on smithi123 with /dev/vg_nvme/lv_3... 2024-08-19T01:26:54.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-19T01:26:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:54.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:26:54.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:54.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:55.120 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:26:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:55 smithi064 ceph-mon[18515]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-19T01:26:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:55 smithi064 ceph-mon[18515]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T01:26:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:55 smithi123 ceph-mon[21829]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-19T01:26:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:55 smithi123 ceph-mon[21829]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T01:26:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:55 smithi161 ceph-mon[21757]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-19T01:26:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:55 smithi161 ceph-mon[21757]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T01:26:56.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:56 smithi123 ceph-mon[21829]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-19T01:26:56.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:56 smithi123 ceph-mon[21829]: osdmap e22: 4 total, 3 up, 4 in 2024-08-19T01:26:56.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:56 smithi123 ceph-mon[21829]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:26:56.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:56 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:56.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:56 smithi123 ceph-mon[21829]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:26:56.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:56 smithi123 ceph-mon[21829]: osdmap e23: 4 total, 3 up, 4 in 2024-08-19T01:26:56.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-mon[18515]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-19T01:26:56.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-mon[18515]: osdmap e22: 4 total, 3 up, 4 in 2024-08-19T01:26:56.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-mon[18515]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:26:56.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:56.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-mon[18515]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:26:56.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-mon[18515]: osdmap e23: 4 total, 3 up, 4 in 2024-08-19T01:26:56.736 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:26:56.500+0000 7fec16bac640 -1 osd.3 0 waiting for initial osdmap 2024-08-19T01:26:56.736 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:26:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:26:56.505+0000 7fec129d4640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:26:56.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:56 smithi161 ceph-mon[21757]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-19T01:26:56.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:56 smithi161 ceph-mon[21757]: osdmap e22: 4 total, 3 up, 4 in 2024-08-19T01:26:56.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:56 smithi161 ceph-mon[21757]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:26:56.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:56 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:56.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:56 smithi161 ceph-mon[21757]: from='osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:26:56.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:56 smithi161 ceph-mon[21757]: osdmap e23: 4 total, 3 up, 4 in 2024-08-19T01:26:57.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:57 smithi064 ceph-mon[18515]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:57.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:57 smithi064 ceph-mon[18515]: osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] boot 2024-08-19T01:26:57.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:57 smithi064 ceph-mon[18515]: osdmap e24: 4 total, 4 up, 4 in 2024-08-19T01:26:57.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:57 smithi123 ceph-mon[21829]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:57 smithi123 ceph-mon[21829]: osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] boot 2024-08-19T01:26:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:57 smithi123 ceph-mon[21829]: osdmap e24: 4 total, 4 up, 4 in 2024-08-19T01:26:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:26:57.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi123:vg_nvme/lv_3 2024-08-19T01:26:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:57 smithi161 ceph-mon[21757]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-08-19T01:26:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:57.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:57 smithi161 ceph-mon[21757]: osd.3 [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] boot 2024-08-19T01:26:57.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:57 smithi161 ceph-mon[21757]: osdmap e24: 4 total, 4 up, 4 in 2024-08-19T01:26:57.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:26:58.025 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 -- 172.21.15.123:0/1054839226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63101076c0 msgr2=0x7f6310109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 --2- 172.21.15.123:0/1054839226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63101076c0 0x7f6310109ab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6308009ca0 tx=0x7f630802a8c0 comp rx=0 tx=0).stop 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 -- 172.21.15.123:0/1054839226 shutdown_connections 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 --2- 172.21.15.123:0/1054839226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63101076c0 0x7f6310109ab0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 --2- 172.21.15.123:0/1054839226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6310104d90 0x7f6310107180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 --2- 172.21.15.123:0/1054839226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6310102460 0x7f6310104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 -- 172.21.15.123:0/1054839226 >> 172.21.15.123:0/1054839226 conn(0x7f63100fbfa0 msgr2=0x7f63100fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 -- 172.21.15.123:0/1054839226 shutdown_connections 2024-08-19T01:26:58.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 -- 172.21.15.123:0/1054839226 wait complete. 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 Processor -- start 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.480+0000 7f6316fc0640 1 -- start start 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6316fc0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6310102460 0x7f6310101dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6316fc0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6310104d90 0x7f6310100410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6316fc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63101076c0 0x7f6310100950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6316fc0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f631010d5c0 con 0x7f6310102460 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6316fc0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f631010d440 con 0x7f6310104d90 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6316fc0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f631010d740 con 0x7f63101076c0 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6310104d90 0x7f6310100410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:58.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6310104d90 0x7f6310100410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34208/0 (socket says 172.21.15.123:34208) 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 -- 172.21.15.123:0/3585516546 learned_addr learned my addr 172.21.15.123:0/3585516546 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6314d5c640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6310102460 0x7f6310101dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 -- 172.21.15.123:0/3585516546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63101076c0 msgr2=0x7f6310100950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63101076c0 0x7f6310100950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 -- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6310102460 msgr2=0x7f6310101dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6310102460 0x7f6310101dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 -- 172.21.15.123:0/3585516546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6310101210 con 0x7f6310104d90 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630ffff640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6310104d90 0x7f6310100410 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f630000d770 tx=0x7f630000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630dffb640 1 -- 172.21.15.123:0/3585516546 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300017070 con 0x7f6310104d90 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63101a1d80 con 0x7f6310104d90 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630dffb640 1 -- 172.21.15.123:0/3585516546 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6300009c30 con 0x7f6310104d90 2024-08-19T01:26:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.481+0000 7f630dffb640 1 -- 172.21.15.123:0/3585516546 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630000f620 con 0x7f6310104d90 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.482+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63101a2240 con 0x7f6310104d90 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.483+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63100fda90 con 0x7f6310104d90 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.483+0000 7f630dffb640 1 -- 172.21.15.123:0/3585516546 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f630001a070 con 0x7f6310104d90 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.484+0000 7f630dffb640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f62fc0a2b80 0x7f62fc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.485+0000 7f630dffb640 1 -- 172.21.15.123:0/3585516546 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 3702+0+0 (secure 0 0 0) 0x7f63000cdd70 con 0x7f6310104d90 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.485+0000 7f6314d5c640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f62fc0a2b80 0x7f62fc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.485+0000 7f6314d5c640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f62fc0a2b80 0x7f62fc0a5040 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62f8005e10 tx=0x7f62f8005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:26:58.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.489+0000 7f630dffb640 1 -- 172.21.15.123:0/3585516546 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6300009de0 con 0x7f6310104d90 2024-08-19T01:26:58.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:26:58.666+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6310062360 con 0x7f62fc0a2b80 2024-08-19T01:26:58.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:26:58.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:26:58.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi064 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi064 to 8544M 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:58 smithi161 ceph-mon[21757]: osdmap e25: 4 total, 4 up, 4 in 2024-08-19T01:26:58.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:26:58.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:26:58.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:26:58.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi064 2024-08-19T01:26:58.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:58.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:58.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi064 to 8544M 2024-08-19T01:26:58.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:58.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:58.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:58.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:58 smithi064 ceph-mon[18515]: osdmap e25: 4 total, 4 up, 4 in 2024-08-19T01:26:59.078 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:26:59.078 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi064 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi064 to 8544M 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:26:59.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:58 smithi123 ceph-mon[21829]: osdmap e25: 4 total, 4 up, 4 in 2024-08-19T01:26:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:59 smithi161 ceph-mon[21757]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:59 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:59.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:59 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:59.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:59 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:59.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:26:59 smithi161 ceph-mon[21757]: osdmap e26: 4 total, 4 up, 4 in 2024-08-19T01:26:59.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:59 smithi064 ceph-mon[18515]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:26:59.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:59 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:26:59.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:59 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:26:59.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:59 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:26:59.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:26:59 smithi064 ceph-mon[18515]: osdmap e26: 4 total, 4 up, 4 in 2024-08-19T01:27:00.110 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:59 smithi123 ceph-mon[21829]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:27:00.110 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:59 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:27:00.110 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:59 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:27:00.111 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:59 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:00.111 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:26:59 smithi123 ceph-mon[21829]: osdmap e26: 4 total, 4 up, 4 in 2024-08-19T01:27:01.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:01 smithi123 ceph-mon[21829]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:01.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:01 smithi123 ceph-mon[21829]: osdmap e27: 4 total, 4 up, 4 in 2024-08-19T01:27:01.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:01 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/3057140271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]: dispatch 2024-08-19T01:27:01.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:01 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]: dispatch 2024-08-19T01:27:01.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:01 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]': finished 2024-08-19T01:27:01.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:01 smithi123 ceph-mon[21829]: osdmap e28: 5 total, 4 up, 5 in 2024-08-19T01:27:01.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:01.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:01 smithi161 ceph-mon[21757]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:01.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:01 smithi161 ceph-mon[21757]: osdmap e27: 4 total, 4 up, 4 in 2024-08-19T01:27:01.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:01 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/3057140271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]: dispatch 2024-08-19T01:27:01.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:01 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]: dispatch 2024-08-19T01:27:01.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:01 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]': finished 2024-08-19T01:27:01.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:01 smithi161 ceph-mon[21757]: osdmap e28: 5 total, 4 up, 5 in 2024-08-19T01:27:01.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:01.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:01 smithi064 ceph-mon[18515]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:01.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:01 smithi064 ceph-mon[18515]: osdmap e27: 4 total, 4 up, 4 in 2024-08-19T01:27:01.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:01 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/3057140271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]: dispatch 2024-08-19T01:27:01.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:01 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]: dispatch 2024-08-19T01:27:01.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:01 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41"}]': finished 2024-08-19T01:27:01.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:01 smithi064 ceph-mon[18515]: osdmap e28: 5 total, 4 up, 5 in 2024-08-19T01:27:01.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:02.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:02 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/3342618124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:02 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/3342618124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:02 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/3342618124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:03.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:03 smithi123 ceph-mon[21829]: pgmap v92: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-08-19T01:27:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:03 smithi161 ceph-mon[21757]: pgmap v92: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-08-19T01:27:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:03 smithi064 ceph-mon[18515]: pgmap v92: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-08-19T01:27:05.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:05 smithi123 ceph-mon[21829]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-08-19T01:27:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:05 smithi161 ceph-mon[21757]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-08-19T01:27:05.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:05 smithi064 ceph-mon[18515]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-08-19T01:27:07.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:07 smithi123 ceph-mon[21829]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-19T01:27:07.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:07 smithi161 ceph-mon[21757]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-19T01:27:07.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:07 smithi064 ceph-mon[18515]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-19T01:27:08.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:08 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-19T01:27:08.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:08 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:08.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:08 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-19T01:27:08.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:08 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:08.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:08 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-19T01:27:08.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:08 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:09.831 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:09 smithi123 ceph-mon[21829]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T01:27:09.832 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:09 smithi123 ceph-mon[21829]: Deploying daemon osd.4 on smithi123 2024-08-19T01:27:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:09 smithi161 ceph-mon[21757]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T01:27:09.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:09 smithi161 ceph-mon[21757]: Deploying daemon osd.4 on smithi123 2024-08-19T01:27:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:09 smithi064 ceph-mon[18515]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T01:27:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:09 smithi064 ceph-mon[18515]: Deploying daemon osd.4 on smithi123 2024-08-19T01:27:11.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:11 smithi123 ceph-mon[21829]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:11 smithi161 ceph-mon[21757]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:11.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:11 smithi064 ceph-mon[18515]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:13.814 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:13 smithi123 ceph-mon[21829]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:13.814 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:13.814 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:13.814 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:13 smithi161 ceph-mon[21757]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:13.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:13 smithi064 ceph-mon[18515]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:13.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:13.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:13.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:15.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:15 smithi123 ceph-mon[21829]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:15 smithi161 ceph-mon[21757]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:15.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:15 smithi064 ceph-mon[18515]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:16.609 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 4 on host 'smithi123' 2024-08-19T01:27:16.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.606+0000 7f630dffb640 1 -- 172.21.15.123:0/3585516546 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6310062360 con 0x7f62fc0a2b80 2024-08-19T01:27:16.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.608+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f62fc0a2b80 msgr2=0x7f62fc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:16.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.608+0000 7f6316fc0640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f62fc0a2b80 0x7f62fc0a5040 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62f8005e10 tx=0x7f62f8005da0 comp rx=0 tx=0).stop 2024-08-19T01:27:16.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6310104d90 msgr2=0x7f6310100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:16.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6310104d90 0x7f6310100410 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f630000d770 tx=0x7f630000dc30 comp rx=0 tx=0).stop 2024-08-19T01:27:16.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 shutdown_connections 2024-08-19T01:27:16.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f62fc0a2b80 0x7f62fc0a5040 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:16.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63101076c0 0x7f6310100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:16.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6310104d90 0x7f6310100410 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:16.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 --2- 172.21.15.123:0/3585516546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6310102460 0x7f6310101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:16.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 >> 172.21.15.123:0/3585516546 conn(0x7f63100fbfa0 msgr2=0x7f6310102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:27:16.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 shutdown_connections 2024-08-19T01:27:16.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:27:16.609+0000 7f6316fc0640 1 -- 172.21.15.123:0/3585516546 wait complete. 2024-08-19T01:27:17.309 DEBUG:teuthology.orchestra.run.smithi123:osd.4> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.4.service 2024-08-19T01:27:17.311 INFO:tasks.cephadm:Deploying osd.5 on smithi161 with /dev/vg_nvme/lv_3... 2024-08-19T01:27:17.311 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-19T01:27:17.571 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:27:17.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:17 smithi161 ceph-mon[21757]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:17.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:17 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:17 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:17 smithi161 ceph-mon[21757]: from='osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T01:27:17.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:17 smithi161 ceph-mon[21757]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T01:27:17.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:17 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:17 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:17 smithi123 ceph-mon[21829]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:17 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:17 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:17 smithi123 ceph-mon[21829]: from='osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T01:27:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:17 smithi123 ceph-mon[21829]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T01:27:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:17 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:17 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:17 smithi064 ceph-mon[18515]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:17 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:17 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:17 smithi064 ceph-mon[18515]: from='osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T01:27:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:17 smithi064 ceph-mon[18515]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T01:27:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:17 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:17 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:18.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:18 smithi123 ceph-mon[21829]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-19T01:27:18.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:18 smithi123 ceph-mon[21829]: from='osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:27:18.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:18 smithi123 ceph-mon[21829]: osdmap e29: 5 total, 4 up, 5 in 2024-08-19T01:27:18.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:18 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:18.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:18 smithi123 ceph-mon[21829]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:27:18.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:18 smithi161 ceph-mon[21757]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-19T01:27:18.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:18 smithi161 ceph-mon[21757]: from='osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:27:18.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:18 smithi161 ceph-mon[21757]: osdmap e29: 5 total, 4 up, 5 in 2024-08-19T01:27:18.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:18 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:18.857 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:18 smithi161 ceph-mon[21757]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:27:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:18 smithi064 ceph-mon[18515]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-19T01:27:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:18 smithi064 ceph-mon[18515]: from='osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:27:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:18 smithi064 ceph-mon[18515]: osdmap e29: 5 total, 4 up, 5 in 2024-08-19T01:27:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:18 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:18 smithi064 ceph-mon[18515]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:27:19.352 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:27:19.021+0000 7fb398a01640 -1 osd.4 0 waiting for initial osdmap 2024-08-19T01:27:19.352 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:27:19.027+0000 7fb390821640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:27:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-mon[21829]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-mon[21829]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:27:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-mon[21829]: osdmap e30: 5 total, 4 up, 5 in 2024-08-19T01:27:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:19 smithi123 ceph-mon[21829]: from='osd.4 ' entity='osd.4' 2024-08-19T01:27:19.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:19 smithi161 ceph-mon[21757]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:19.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:19 smithi161 ceph-mon[21757]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:27:19.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:19 smithi161 ceph-mon[21757]: osdmap e30: 5 total, 4 up, 5 in 2024-08-19T01:27:19.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:19 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:19.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:19 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:19.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:19 smithi161 ceph-mon[21757]: from='osd.4 ' entity='osd.4' 2024-08-19T01:27:19.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:19 smithi064 ceph-mon[18515]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:19.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:19 smithi064 ceph-mon[18515]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:27:19.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:19 smithi064 ceph-mon[18515]: osdmap e30: 5 total, 4 up, 5 in 2024-08-19T01:27:19.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:19 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:19.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:19 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:19.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:19 smithi064 ceph-mon[18515]: from='osd.4 ' entity='osd.4' 2024-08-19T01:27:20.257 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:27:20.283 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_3 2024-08-19T01:27:20.542 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:27:20.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:20 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:27:20.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:20 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:27:20.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:20.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:20 smithi161 ceph-mon[21757]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:20.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:20 smithi161 ceph-mon[21757]: osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] boot 2024-08-19T01:27:20.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:20 smithi161 ceph-mon[21757]: osdmap e31: 5 total, 5 up, 5 in 2024-08-19T01:27:20.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:20 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:27:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:20 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:27:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:20 smithi123 ceph-mon[21829]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:20 smithi123 ceph-mon[21829]: osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] boot 2024-08-19T01:27:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:20 smithi123 ceph-mon[21829]: osdmap e31: 5 total, 5 up, 5 in 2024-08-19T01:27:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:20.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:20 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:27:20.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:20 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:27:20.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:20.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:20 smithi064 ceph-mon[18515]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-08-19T01:27:20.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:20 smithi064 ceph-mon[18515]: osd.4 [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] boot 2024-08-19T01:27:20.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:20 smithi064 ceph-mon[18515]: osdmap e31: 5 total, 5 up, 5 in 2024-08-19T01:27:20.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:27:21.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 -- 172.21.15.161:0/714657484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6741075f0 msgr2=0x7fd674109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:21.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 --2- 172.21.15.161:0/714657484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6741075f0 0x7fd674109a40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009ca0 tx=0x7fd66c02a8c0 comp rx=0 tx=0).stop 2024-08-19T01:27:21.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 -- 172.21.15.161:0/714657484 shutdown_connections 2024-08-19T01:27:21.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 --2- 172.21.15.161:0/714657484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6741075f0 0x7fd674109a40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 --2- 172.21.15.161:0/714657484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd674104cc0 0x7fd6741070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 --2- 172.21.15.161:0/714657484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd674102390 0x7fd674104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 -- 172.21.15.161:0/714657484 >> 172.21.15.161:0/714657484 conn(0x7fd6740fbfb0 msgr2=0x7fd6740fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 -- 172.21.15.161:0/714657484 shutdown_connections 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.022+0000 7fd67b90a640 1 -- 172.21.15.161:0/714657484 wait complete. 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 Processor -- start 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 -- start start 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd674102390 0x7fd674101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd674104cc0 0x7fd674100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741075f0 0x7fd674100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd67410d5e0 con 0x7fd674102390 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd67410d460 con 0x7fd674104cc0 2024-08-19T01:27:21.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd67b90a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd67410d760 con 0x7fd6741075f0 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd679ea7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741075f0 0x7fd674100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd679ea7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741075f0 0x7fd674100880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40564/0 (socket says 172.21.15.161:40564) 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd679ea7640 1 -- 172.21.15.161:0/1934579082 learned_addr learned my addr 172.21.15.161:0/1934579082 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.023+0000 7fd678ea5640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd674104cc0 0x7fd674100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd6796a6640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd674102390 0x7fd674101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd678ea5640 1 -- 172.21.15.161:0/1934579082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741075f0 msgr2=0x7fd674100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd678ea5640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741075f0 0x7fd674100880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd678ea5640 1 -- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd674102390 msgr2=0x7fd674101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd678ea5640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd674102390 0x7fd674101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd678ea5640 1 -- 172.21.15.161:0/1934579082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd674101140 con 0x7fd674104cc0 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd6796a6640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd674102390 0x7fd674101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd679ea7640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741075f0 0x7fd674100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.024+0000 7fd678ea5640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd674104cc0 0x7fd674100340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd67000e9c0 tx=0x7fd67000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:27:21.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.025+0000 7fd66a7fc640 1 -- 172.21.15.161:0/1934579082 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67000cd80 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.025+0000 7fd66a7fc640 1 -- 172.21.15.161:0/1934579082 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6700035f0 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.025+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6741a1d70 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.025+0000 7fd66a7fc640 1 -- 172.21.15.161:0/1934579082 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670010690 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.025+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6741a2270 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.026+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6740fdaa0 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.027+0000 7fd66a7fc640 1 -- 172.21.15.161:0/1934579082 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd670004020 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.028+0000 7fd66a7fc640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd6600a2c00 0x7fd6600a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.028+0000 7fd66a7fc640 1 -- 172.21.15.161:0/1934579082 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4101+0+0 (secure 0 0 0) 0x7fd6700ce330 con 0x7fd674104cc0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.028+0000 7fd6796a6640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd6600a2c00 0x7fd6600a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.029+0000 7fd6796a6640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd6600a2c00 0x7fd6600a50c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd664009c30 tx=0x7fd664005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:27:21.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.032+0000 7fd66a7fc640 1 -- 172.21.15.161:0/1934579082 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd67009c260 con 0x7fd674104cc0 2024-08-19T01:27:21.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:21.223+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd674062360 con 0x7fd6600a2c00 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi123 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi123 to 8544M 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: osdmap e32: 5 total, 5 up, 5 in 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:22.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:22.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:27:22.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:27:22.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:27:22.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi123 2024-08-19T01:27:22.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:22.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:22.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi123 to 8544M 2024-08-19T01:27:22.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: osdmap e32: 5 total, 5 up, 5 in 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:27:22.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi123 2024-08-19T01:27:22.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:22.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:22.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi123 to 8544M 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: osdmap e32: 5 total, 5 up, 5 in 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:27:22.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:23.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:23 smithi161 ceph-mon[21757]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:23.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:23 smithi064 ceph-mon[18515]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:23.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:23 smithi123 ceph-mon[21829]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:24.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:24 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/1250710252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]: dispatch 2024-08-19T01:27:24.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:24 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]: dispatch 2024-08-19T01:27:24.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:24 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]': finished 2024-08-19T01:27:24.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:24 smithi161 ceph-mon[21757]: osdmap e33: 6 total, 5 up, 6 in 2024-08-19T01:27:24.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:24.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:24 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/108534838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:24.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:24 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/1250710252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]: dispatch 2024-08-19T01:27:24.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:24 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]: dispatch 2024-08-19T01:27:24.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:24 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]': finished 2024-08-19T01:27:24.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:24 smithi064 ceph-mon[18515]: osdmap e33: 6 total, 5 up, 6 in 2024-08-19T01:27:24.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:24.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:24 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/108534838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:24.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:24 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/1250710252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]: dispatch 2024-08-19T01:27:24.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:24 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]: dispatch 2024-08-19T01:27:24.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:24 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3566d78c-0daf-4391-bd31-1f8c95f93d35"}]': finished 2024-08-19T01:27:24.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:24 smithi123 ceph-mon[21829]: osdmap e33: 6 total, 5 up, 6 in 2024-08-19T01:27:24.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:24.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:24 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/108534838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:25 smithi123 ceph-mon[21829]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:25.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:25 smithi161 ceph-mon[21757]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:25.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:25 smithi064 ceph-mon[18515]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:27.847 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:27 smithi161 ceph-mon[21757]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:27.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:27 smithi123 ceph-mon[21829]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:27.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:27 smithi064 ceph-mon[18515]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:29.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:29 smithi123 ceph-mon[21829]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:29.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:29 smithi161 ceph-mon[21757]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:29.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:29 smithi064 ceph-mon[18515]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:31.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:31 smithi161 ceph-mon[21757]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:31.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-19T01:27:31.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:31.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:31 smithi123 ceph-mon[21829]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:31.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-19T01:27:31.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:31 smithi064 ceph-mon[18515]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-19T01:27:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:32 smithi123 ceph-mon[21829]: Deploying daemon osd.5 on smithi161 2024-08-19T01:27:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:32 smithi161 ceph-mon[21757]: Deploying daemon osd.5 on smithi161 2024-08-19T01:27:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:32 smithi064 ceph-mon[18515]: Deploying daemon osd.5 on smithi161 2024-08-19T01:27:33.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:33 smithi123 ceph-mon[21829]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:33.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:33 smithi161 ceph-mon[21757]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:33.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:33 smithi064 ceph-mon[18515]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:34.830 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:34 smithi161 ceph-mon[21757]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:34.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:34 smithi123 ceph-mon[21829]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:34.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:34 smithi064 ceph-mon[18515]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:36.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:36 smithi161 ceph-mon[21757]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:36.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:36.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:36.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:36 smithi064 ceph-mon[18515]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:36.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:37.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:36 smithi123 ceph-mon[21829]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:37.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:37.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:37.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:38.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:38 smithi161 ceph-mon[21757]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:38.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:38.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:38.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:38.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:38.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:38 smithi064 ceph-mon[18515]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:38 smithi123 ceph-mon[21829]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:39.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:39.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:39.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.305+0000 7fd66a7fc640 1 -- 172.21.15.161:0/1934579082 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd674062360 con 0x7fd6600a2c00 2024-08-19T01:27:39.308 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 5 on host 'smithi161' 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd6600a2c00 msgr2=0x7fd6600a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd6600a2c00 0x7fd6600a50c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd664009c30 tx=0x7fd664005f90 comp rx=0 tx=0).stop 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd674104cc0 msgr2=0x7fd674100340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd674104cc0 0x7fd674100340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd67000e9c0 tx=0x7fd67000ee80 comp rx=0 tx=0).stop 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 shutdown_connections 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd6600a2c00 0x7fd6600a50c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741075f0 0x7fd674100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:39.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd674104cc0 0x7fd674100340 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:39.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 --2- 172.21.15.161:0/1934579082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd674102390 0x7fd674101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:39.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 >> 172.21.15.161:0/1934579082 conn(0x7fd6740fbfb0 msgr2=0x7fd674102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:27:39.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 shutdown_connections 2024-08-19T01:27:39.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:27:39.308+0000 7fd67b90a640 1 -- 172.21.15.161:0/1934579082 wait complete. 2024-08-19T01:27:39.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:39.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:39 smithi161 ceph-mon[21757]: from='osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T01:27:39.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:39 smithi161 ceph-mon[21757]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T01:27:39.915 DEBUG:teuthology.orchestra.run.smithi161:osd.5> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.5.service 2024-08-19T01:27:39.917 INFO:tasks.cephadm:Deploying osd.6 on smithi064 with /dev/vg_nvme/lv_2... 2024-08-19T01:27:39.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-19T01:27:39.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:39.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:39.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:39 smithi064 ceph-mon[18515]: from='osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T01:27:39.946 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:39 smithi064 ceph-mon[18515]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T01:27:40.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:40.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:40.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:40.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:39 smithi123 ceph-mon[21829]: from='osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T01:27:40.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:39 smithi123 ceph-mon[21829]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T01:27:40.186 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:27:41.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-mon[21757]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:41.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-mon[21757]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-19T01:27:41.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-mon[21757]: osdmap e34: 6 total, 5 up, 6 in 2024-08-19T01:27:41.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:41.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-mon[21757]: from='osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:27:41.451 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-mon[21757]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:27:41.451 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:27:41.316+0000 7f79f5f0b640 -1 osd.5 0 waiting for initial osdmap 2024-08-19T01:27:41.451 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:27:41 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:27:41.322+0000 7f79f1d33640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:27:41.603 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:41 smithi064 ceph-mon[18515]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:41.603 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:41 smithi064 ceph-mon[18515]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-19T01:27:41.603 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:41 smithi064 ceph-mon[18515]: osdmap e34: 6 total, 5 up, 6 in 2024-08-19T01:27:41.603 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:41 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:41.603 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:41 smithi064 ceph-mon[18515]: from='osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:27:41.603 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:41 smithi064 ceph-mon[18515]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:27:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:41 smithi123 ceph-mon[21829]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:41 smithi123 ceph-mon[21829]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-19T01:27:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:41 smithi123 ceph-mon[21829]: osdmap e34: 6 total, 5 up, 6 in 2024-08-19T01:27:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:41 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:41 smithi123 ceph-mon[21829]: from='osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:27:41.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:41 smithi123 ceph-mon[21829]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:27:42.501 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:42 smithi161 ceph-mon[21757]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:27:42.501 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:42 smithi161 ceph-mon[21757]: osdmap e35: 6 total, 5 up, 6 in 2024-08-19T01:27:42.501 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:42.501 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:42.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:42 smithi123 ceph-mon[21829]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:27:42.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:42 smithi123 ceph-mon[21829]: osdmap e35: 6 total, 5 up, 6 in 2024-08-19T01:27:42.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:42.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:42.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:42 smithi064 ceph-mon[18515]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:27:42.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:42 smithi064 ceph-mon[18515]: osdmap e35: 6 total, 5 up, 6 in 2024-08-19T01:27:42.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:42.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:42.702 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:27:42.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_2 2024-08-19T01:27:43.029 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:27:43.411 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] boot 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: osdmap e36: 6 total, 6 up, 6 in 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:43.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:43.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:43.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:43.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:43 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 -- 172.21.15.64:0/562990650 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c101930 msgr2=0x7fb89c101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/562990650 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c101930 0x7fb89c101d30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009900 tx=0x7fb88c02a410 comp rx=0 tx=0).stop 2024-08-19T01:27:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 -- 172.21.15.64:0/562990650 shutdown_connections 2024-08-19T01:27:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/562990650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89c1034f0 0x7fb89c109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/562990650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb89c102b30 0x7fb89c102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/562990650 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c101930 0x7fb89c101d30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 -- 172.21.15.64:0/562990650 >> 172.21.15.64:0/562990650 conn(0x7fb89c0fd0c0 msgr2=0x7fb89c0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:27:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 -- 172.21.15.64:0/562990650 shutdown_connections 2024-08-19T01:27:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.494+0000 7fb8a1a23640 1 -- 172.21.15.64:0/562990650 wait complete. 2024-08-19T01:27:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 Processor -- start 2024-08-19T01:27:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 -- start start 2024-08-19T01:27:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb89c101930 0x7fb89c197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c102b30 0x7fb89c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89c1034f0 0x7fb89c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c10d5b0 con 0x7fb89c102b30 2024-08-19T01:27:43.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c10d430 con 0x7fb89c1034f0 2024-08-19T01:27:43.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb8a1a23640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c10d730 con 0x7fb89c101930 2024-08-19T01:27:43.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb893fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c102b30 0x7fb89c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:43.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb893fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c102b30 0x7fb89c198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44118/0 (socket says 172.21.15.64:44118) 2024-08-19T01:27:43.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb893fff640 1 -- 172.21.15.64:0/1424151029 learned_addr learned my addr 172.21.15.64:0/1424151029 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:27:43.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb89affd640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb89c101930 0x7fb89c197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:43.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.495+0000 7fb89b7fe640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89c1034f0 0x7fb89c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:43.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb893fff640 1 -- 172.21.15.64:0/1424151029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb89c101930 msgr2=0x7fb89c197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:43.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb893fff640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb89c101930 0x7fb89c197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:43.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb893fff640 1 -- 172.21.15.64:0/1424151029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89c1034f0 msgr2=0x7fb89c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:27:43.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb893fff640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89c1034f0 0x7fb89c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:27:43.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb893fff640 1 -- 172.21.15.64:0/1424151029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89c19f840 con 0x7fb89c102b30 2024-08-19T01:27:43.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb893fff640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c102b30 0x7fb89c198160 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c970 tx=0x7fb88400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:27:43.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb8a0a21640 1 -- 172.21.15.64:0/1424151029 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884007bb0 con 0x7fb89c102b30 2024-08-19T01:27:43.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89c19fb30 con 0x7fb89c102b30 2024-08-19T01:27:43.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb8a0a21640 1 -- 172.21.15.64:0/1424151029 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb884007d50 con 0x7fb89c102b30 2024-08-19T01:27:43.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb8a0a21640 1 -- 172.21.15.64:0/1424151029 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88400f450 con 0x7fb89c102b30 2024-08-19T01:27:43.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.496+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89c1a0040 con 0x7fb89c102b30 2024-08-19T01:27:43.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.497+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb89c108d20 con 0x7fb89c102b30 2024-08-19T01:27:43.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.498+0000 7fb8a0a21640 1 -- 172.21.15.64:0/1424151029 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb884004fe0 con 0x7fb89c102b30 2024-08-19T01:27:43.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.499+0000 7fb8a0a21640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb8880a2b80 0x7fb8880a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:27:43.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.499+0000 7fb89affd640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb8880a2b80 0x7fb8880a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:27:43.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.499+0000 7fb8a0a21640 1 -- 172.21.15.64:0/1424151029 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4562+0+0 (secure 0 0 0) 0x7fb8840ce2c0 con 0x7fb89c102b30 2024-08-19T01:27:43.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.499+0000 7fb89affd640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb8880a2b80 0x7fb8880a5040 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb88c005ab0 tx=0x7fb88c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:27:43.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.502+0000 7fb8a0a21640 1 -- 172.21.15.64:0/1424151029 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb884014030 con 0x7fb89c102b30 2024-08-19T01:27:43.587 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:27:43.587 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:27:43.587 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:43.587 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] boot 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: osdmap e36: 6 total, 6 up, 6 in 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:43 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:27:43.688+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb89c062360 con 0x7fb8880a2b80 2024-08-19T01:27:43.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:27:43.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: osd.5 [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] boot 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: osdmap e36: 6 total, 6 up, 6 in 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:27:43.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:43 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:44.406 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:44 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi161 2024-08-19T01:27:44.406 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:44 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi161 to 8544M 2024-08-19T01:27:44.406 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:44 smithi064 ceph-mon[18515]: osdmap e37: 6 total, 6 up, 6 in 2024-08-19T01:27:44.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:27:44.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:27:44.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:44 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi161 2024-08-19T01:27:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:44 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi161 to 8544M 2024-08-19T01:27:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:44 smithi123 ceph-mon[21829]: osdmap e37: 6 total, 6 up, 6 in 2024-08-19T01:27:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:27:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:27:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:44.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:44 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi161 2024-08-19T01:27:44.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:44 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi161 to 8544M 2024-08-19T01:27:44.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:44 smithi161 ceph-mon[21757]: osdmap e37: 6 total, 6 up, 6 in 2024-08-19T01:27:44.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:27:44.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:27:44.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:45.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:45 smithi064 ceph-mon[18515]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:45.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:45 smithi064 ceph-mon[18515]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:27:45.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:45 smithi064 ceph-mon[18515]: osdmap e38: 6 total, 6 up, 6 in 2024-08-19T01:27:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:45 smithi123 ceph-mon[21829]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:45 smithi123 ceph-mon[21829]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:27:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:45 smithi123 ceph-mon[21829]: osdmap e38: 6 total, 6 up, 6 in 2024-08-19T01:27:45.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:45 smithi161 ceph-mon[21757]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:45 smithi161 ceph-mon[21757]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:27:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:45 smithi161 ceph-mon[21757]: osdmap e38: 6 total, 6 up, 6 in 2024-08-19T01:27:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:46 smithi123 ceph-mon[21829]: osdmap e39: 6 total, 6 up, 6 in 2024-08-19T01:27:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:46 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3549028999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727df6d7-b958-434a-9812-a1b798a99b8f"}]: dispatch 2024-08-19T01:27:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:46 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3549028999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "727df6d7-b958-434a-9812-a1b798a99b8f"}]': finished 2024-08-19T01:27:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:46 smithi123 ceph-mon[21829]: osdmap e40: 7 total, 6 up, 7 in 2024-08-19T01:27:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:46 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:27:46.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:46 smithi161 ceph-mon[21757]: osdmap e39: 6 total, 6 up, 6 in 2024-08-19T01:27:46.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:46 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3549028999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727df6d7-b958-434a-9812-a1b798a99b8f"}]: dispatch 2024-08-19T01:27:46.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:46 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3549028999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "727df6d7-b958-434a-9812-a1b798a99b8f"}]': finished 2024-08-19T01:27:46.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:46 smithi161 ceph-mon[21757]: osdmap e40: 7 total, 6 up, 7 in 2024-08-19T01:27:46.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:46 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:27:46.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:46 smithi064 ceph-mon[18515]: osdmap e39: 6 total, 6 up, 6 in 2024-08-19T01:27:46.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:46 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3549028999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727df6d7-b958-434a-9812-a1b798a99b8f"}]: dispatch 2024-08-19T01:27:46.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:46 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3549028999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "727df6d7-b958-434a-9812-a1b798a99b8f"}]': finished 2024-08-19T01:27:46.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:46 smithi064 ceph-mon[18515]: osdmap e40: 7 total, 6 up, 7 in 2024-08-19T01:27:46.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:46 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:27:47.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:47 smithi123 ceph-mon[21829]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-08-19T01:27:47.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:47 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3819659831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:47.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:47 smithi161 ceph-mon[21757]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-08-19T01:27:47.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:47 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3819659831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:47.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:47 smithi064 ceph-mon[18515]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-08-19T01:27:47.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:47 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3819659831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:27:49.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:49 smithi123 ceph-mon[21829]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-08-19T01:27:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:49 smithi161 ceph-mon[21757]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-08-19T01:27:49.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:49 smithi064 ceph-mon[18515]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-08-19T01:27:51.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:51 smithi064 ceph-mon[18515]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-19T01:27:51.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:51 smithi123 ceph-mon[21829]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-19T01:27:51.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:51 smithi161 ceph-mon[21757]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-19T01:27:53.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:53 smithi064 ceph-mon[18515]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-08-19T01:27:53.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-19T01:27:53.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:53.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:53 smithi123 ceph-mon[21829]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-08-19T01:27:53.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-19T01:27:53.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:53.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:53 smithi161 ceph-mon[21757]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-08-19T01:27:53.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-19T01:27:53.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:27:54.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:54 smithi064 ceph-mon[18515]: Deploying daemon osd.6 on smithi064 2024-08-19T01:27:54.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:54 smithi123 ceph-mon[21829]: Deploying daemon osd.6 on smithi064 2024-08-19T01:27:54.690 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:54 smithi161 ceph-mon[21757]: Deploying daemon osd.6 on smithi064 2024-08-19T01:27:55.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:55 smithi064 ceph-mon[18515]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-19T01:27:55.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:55 smithi123 ceph-mon[21829]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-19T01:27:55.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:55 smithi161 ceph-mon[21757]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-08-19T01:27:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:57 smithi123 ceph-mon[21829]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:57.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:57 smithi161 ceph-mon[21757]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:57.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:57 smithi064 ceph-mon[18515]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:58.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:58.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:58.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:58 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:58.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:58.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:58.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:58 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:58.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:27:58.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:58.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:58 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:27:59.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:27:59 smithi123 ceph-mon[21829]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:59.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:27:59 smithi161 ceph-mon[21757]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:27:59.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:27:59 smithi064 ceph-mon[18515]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:01.569 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:01 smithi064 ceph-mon[18515]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.608+0000 7fb8a0a21640 1 -- 172.21.15.64:0/1424151029 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb89c062360 con 0x7fb8880a2b80 2024-08-19T01:28:01.611 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 6 on host 'smithi064' 2024-08-19T01:28:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb8880a2b80 msgr2=0x7fb8880a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb8880a2b80 0x7fb8880a5040 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb88c005ab0 tx=0x7fb88c000bc0 comp rx=0 tx=0).stop 2024-08-19T01:28:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c102b30 msgr2=0x7fb89c198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c102b30 0x7fb89c198160 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c970 tx=0x7fb88400ce30 comp rx=0 tx=0).stop 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 shutdown_connections 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb8880a2b80 0x7fb8880a5040 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89c1034f0 0x7fb89c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb89c102b30 0x7fb89c198160 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 --2- 172.21.15.64:0/1424151029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb89c101930 0x7fb89c197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 >> 172.21.15.64:0/1424151029 conn(0x7fb89c0fd0c0 msgr2=0x7fb89c105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 shutdown_connections 2024-08-19T01:28:01.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:01.610+0000 7fb8a1a23640 1 -- 172.21.15.64:0/1424151029 wait complete. 2024-08-19T01:28:01.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:01 smithi123 ceph-mon[21829]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:01.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:01 smithi161 ceph-mon[21757]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:02.292 DEBUG:teuthology.orchestra.run.smithi064:osd.6> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.6.service 2024-08-19T01:28:02.295 INFO:tasks.cephadm:Deploying osd.7 on smithi123 with /dev/vg_nvme/lv_2... 2024-08-19T01:28:02.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:28:02.886 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:02 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:02 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:02 smithi123 ceph-mon[21829]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T01:28:02.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:02 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:02 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:02 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:02 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:02 smithi161 ceph-mon[21757]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T01:28:02.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:02 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:02 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:02 smithi064 ceph-mon[18515]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T01:28:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:03.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:03 smithi123 ceph-mon[21829]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:03.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:03 smithi123 ceph-mon[21829]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-19T01:28:03.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:03 smithi123 ceph-mon[21829]: osdmap e41: 7 total, 6 up, 7 in 2024-08-19T01:28:03.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:03 smithi123 ceph-mon[21829]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:28:03.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:03 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:03 smithi161 ceph-mon[21757]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:03 smithi161 ceph-mon[21757]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-19T01:28:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:03 smithi161 ceph-mon[21757]: osdmap e41: 7 total, 6 up, 7 in 2024-08-19T01:28:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:03 smithi161 ceph-mon[21757]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:28:03.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:03 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:03 smithi064 ceph-mon[18515]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:03 smithi064 ceph-mon[18515]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-19T01:28:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:03 smithi064 ceph-mon[18515]: osdmap e41: 7 total, 6 up, 7 in 2024-08-19T01:28:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:03 smithi064 ceph-mon[18515]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:28:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:03 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:03.986 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:28:03 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:28:03.619+0000 7fc59ccd5640 -1 osd.6 0 waiting for initial osdmap 2024-08-19T01:28:03.986 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:28:03 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:28:03.627+0000 7fc598afd640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:28:04.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:04 smithi064 ceph-mon[18515]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:28:04.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:04 smithi064 ceph-mon[18515]: osdmap e42: 7 total, 6 up, 7 in 2024-08-19T01:28:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:04 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:04 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:04.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:04 smithi123 ceph-mon[21829]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:28:04.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:04 smithi123 ceph-mon[21829]: osdmap e42: 7 total, 6 up, 7 in 2024-08-19T01:28:04.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:04 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:04.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:04 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:04.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:04 smithi161 ceph-mon[21757]: from='osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:28:04.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:04 smithi161 ceph-mon[21757]: osdmap e42: 7 total, 6 up, 7 in 2024-08-19T01:28:04.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:04 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:04.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:04 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:05.313 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:28:05.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi123:vg_nvme/lv_2 2024-08-19T01:28:05.606 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:28:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:05 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:28:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:05 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:28:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:05 smithi123 ceph-mon[21829]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:05 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:05 smithi123 ceph-mon[21829]: osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] boot 2024-08-19T01:28:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:05 smithi123 ceph-mon[21829]: osdmap e43: 7 total, 7 up, 7 in 2024-08-19T01:28:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:05 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:05 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:28:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:05 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:28:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:05 smithi161 ceph-mon[21757]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:05.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:05 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:05.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:05 smithi161 ceph-mon[21757]: osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] boot 2024-08-19T01:28:05.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:05 smithi161 ceph-mon[21757]: osdmap e43: 7 total, 7 up, 7 in 2024-08-19T01:28:05.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:05 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:05.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:05 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:28:05.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:05 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:28:05.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:05 smithi064 ceph-mon[18515]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-08-19T01:28:05.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:05 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:05.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:05 smithi064 ceph-mon[18515]: osd.6 [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] boot 2024-08-19T01:28:05.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:05 smithi064 ceph-mon[18515]: osdmap e43: 7 total, 7 up, 7 in 2024-08-19T01:28:05.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:05 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:28:06.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.080+0000 7fa530918640 1 -- 172.21.15.123:0/1248981082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c1023f0 msgr2=0x7fa52c108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:06.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.080+0000 7fa530918640 1 --2- 172.21.15.123:0/1248981082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c1023f0 0x7fa52c108c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa524009ca0 tx=0x7fa52402a8c0 comp rx=0 tx=0).stop 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 -- 172.21.15.123:0/1248981082 shutdown_connections 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 --2- 172.21.15.123:0/1248981082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c1023f0 0x7fa52c108c80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 --2- 172.21.15.123:0/1248981082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa52c101a30 0x7fa52c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 --2- 172.21.15.123:0/1248981082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52c100830 0x7fa52c100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 -- 172.21.15.123:0/1248981082 >> 172.21.15.123:0/1248981082 conn(0x7fa52c0fbfc0 msgr2=0x7fa52c0fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 -- 172.21.15.123:0/1248981082 shutdown_connections 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 -- 172.21.15.123:0/1248981082 wait complete. 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.081+0000 7fa530918640 1 Processor -- start 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa530918640 1 -- start start 2024-08-19T01:28:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa530918640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa52c100830 0x7fa52c06b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa530918640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c101a30 0x7fa52c19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa530918640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52c1023f0 0x7fa52c1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa530918640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10c4b0 con 0x7fa52c100830 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa530918640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10c330 con 0x7fa52c101a30 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa530918640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10c630 con 0x7fa52c1023f0 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c101a30 0x7fa52c19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c101a30 0x7fa52c19ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46548/0 (socket says 172.21.15.123:46548) 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa52a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa52c100830 0x7fa52c06b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 -- 172.21.15.123:0/2318559328 learned_addr learned my addr 172.21.15.123:0/2318559328 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa52ad9d640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52c1023f0 0x7fa52c1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 -- 172.21.15.123:0/2318559328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52c1023f0 msgr2=0x7fa52c1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52c1023f0 0x7fa52c1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:06.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 -- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa52c100830 msgr2=0x7fa52c06b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa52c100830 0x7fa52c06b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa529d9b640 1 -- 172.21.15.123:0/2318559328 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52c1a1a80 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa52ad9d640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52c1023f0 0x7fa52c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.082+0000 7fa52a59c640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa52c100830 0x7fa52c06b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.083+0000 7fa529d9b640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c101a30 0x7fa52c19ed40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa52000d950 tx=0x7fa52000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.083+0000 7fa51b7fe640 1 -- 172.21.15.123:0/2318559328 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520017030 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.083+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52c1a1d10 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.083+0000 7fa51b7fe640 1 -- 172.21.15.123:0/2318559328 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa520004020 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.083+0000 7fa51b7fe640 1 -- 172.21.15.123:0/2318559328 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5200047a0 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.083+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52c1a21f0 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.084+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa52c107c30 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.085+0000 7fa51b7fe640 1 -- 172.21.15.123:0/2318559328 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa5200041d0 con 0x7fa52c101a30 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.086+0000 7fa51b7fe640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5140a2bb0 0x7fa5140a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:06.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.086+0000 7fa51b7fe640 1 -- 172.21.15.123:0/2318559328 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7fa5200cdcd0 con 0x7fa52c101a30 2024-08-19T01:28:06.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.086+0000 7fa52a59c640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5140a2bb0 0x7fa5140a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:06.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.087+0000 7fa52a59c640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5140a2bb0 0x7fa5140a5070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005fd0 tx=0x7fa51c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:28:06.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.094+0000 7fa51b7fe640 1 -- 172.21.15.123:0/2318559328 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa52009b940 con 0x7fa52c101a30 2024-08-19T01:28:06.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:06.279+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa52c062360 con 0x7fa5140a2bb0 2024-08-19T01:28:06.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: osdmap e44: 7 total, 7 up, 7 in 2024-08-19T01:28:06.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:06.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi064 2024-08-19T01:28:06.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi064 to 5696M 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:06.883 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:06.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:06.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: osdmap e44: 7 total, 7 up, 7 in 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi064 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi064 to 5696M 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:06.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:06.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:06.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:06.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:06.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: osdmap e44: 7 total, 7 up, 7 in 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi064 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi064 to 5696M 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:06.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:08.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:08 smithi161 ceph-mon[21757]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:08.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:08 smithi064 ceph-mon[18515]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:09.077 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:08 smithi123 ceph-mon[21829]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:09 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/2636176623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]: dispatch 2024-08-19T01:28:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:09 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]: dispatch 2024-08-19T01:28:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:09 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]': finished 2024-08-19T01:28:09.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:09 smithi161 ceph-mon[21757]: osdmap e45: 8 total, 7 up, 8 in 2024-08-19T01:28:09.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:09.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:09 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/533851146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:09 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/2636176623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]: dispatch 2024-08-19T01:28:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:09 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]: dispatch 2024-08-19T01:28:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:09 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]': finished 2024-08-19T01:28:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:09 smithi064 ceph-mon[18515]: osdmap e45: 8 total, 7 up, 8 in 2024-08-19T01:28:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:09.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:09 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/533851146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:10.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:09 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/2636176623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]: dispatch 2024-08-19T01:28:10.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:09 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]: dispatch 2024-08-19T01:28:10.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:09 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30066775-0caa-4862-bd5b-45711ad1c571"}]': finished 2024-08-19T01:28:10.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:09 smithi123 ceph-mon[21829]: osdmap e45: 8 total, 7 up, 8 in 2024-08-19T01:28:10.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:10.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:09 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/533851146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:11.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:10 smithi123 ceph-mon[21829]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:11.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:10 smithi161 ceph-mon[21757]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:11.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:10 smithi064 ceph-mon[18515]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:13.095 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:12 smithi123 ceph-mon[21829]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:13.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:12 smithi161 ceph-mon[21757]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:13.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:12 smithi064 ceph-mon[18515]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:14.858 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:14 smithi123 ceph-mon[21829]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:15.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:14 smithi161 ceph-mon[21757]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:15.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:14 smithi064 ceph-mon[18515]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:17.041 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:16 smithi123 ceph-mon[21829]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:17.041 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-19T01:28:17.041 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:17.041 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:16 smithi123 ceph-mon[21829]: Deploying daemon osd.7 on smithi123 2024-08-19T01:28:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:16 smithi161 ceph-mon[21757]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-19T01:28:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:16 smithi161 ceph-mon[21757]: Deploying daemon osd.7 on smithi123 2024-08-19T01:28:17.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:16 smithi064 ceph-mon[18515]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:17.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-19T01:28:17.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:17.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:16 smithi064 ceph-mon[18515]: Deploying daemon osd.7 on smithi123 2024-08-19T01:28:19.123 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:18 smithi123 ceph-mon[21829]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:19.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:18 smithi161 ceph-mon[21757]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:19.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:18 smithi064 ceph-mon[18515]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:21.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:20 smithi123 ceph-mon[21829]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:21.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:20 smithi161 ceph-mon[21757]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:21.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:20 smithi064 ceph-mon[18515]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:22.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:21 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:22.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:21 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:22.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:21 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:22.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:21 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:22.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:21 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:22.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:21 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:21 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:22.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:21 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:22.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:21 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:23 smithi123 ceph-mon[21829]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:23.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:23 smithi161 ceph-mon[21757]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:23.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:23 smithi064 ceph-mon[18515]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:24.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.505+0000 7fa51b7fe640 1 -- 172.21.15.123:0/2318559328 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa52c062360 con 0x7fa5140a2bb0 2024-08-19T01:28:24.508 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 7 on host 'smithi123' 2024-08-19T01:28:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5140a2bb0 msgr2=0x7fa5140a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5140a2bb0 0x7fa5140a5070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005fd0 tx=0x7fa51c00b390 comp rx=0 tx=0).stop 2024-08-19T01:28:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c101a30 msgr2=0x7fa52c19ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c101a30 0x7fa52c19ed40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa52000d950 tx=0x7fa52000de10 comp rx=0 tx=0).stop 2024-08-19T01:28:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 shutdown_connections 2024-08-19T01:28:24.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5140a2bb0 0x7fa5140a5070 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52c1023f0 0x7fa52c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa52c101a30 0x7fa52c19ed40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 --2- 172.21.15.123:0/2318559328 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa52c100830 0x7fa52c06b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 >> 172.21.15.123:0/2318559328 conn(0x7fa52c0fbfc0 msgr2=0x7fa52c104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:28:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 shutdown_connections 2024-08-19T01:28:24.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:28:24.507+0000 7fa530918640 1 -- 172.21.15.123:0/2318559328 wait complete. 2024-08-19T01:28:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:25.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:25.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:25.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:25.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:25.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:25.345 DEBUG:teuthology.orchestra.run.smithi123:osd.7> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.7.service 2024-08-19T01:28:25.348 INFO:tasks.cephadm:Deploying osd.8 on smithi161 with /dev/vg_nvme/lv_2... 2024-08-19T01:28:25.348 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-19T01:28:25.607 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:28:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:25 smithi123 ceph-mon[21829]: from='osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T01:28:25.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:25 smithi123 ceph-mon[21829]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T01:28:25.985 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:25 smithi161 ceph-mon[21757]: from='osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T01:28:25.985 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:25 smithi161 ceph-mon[21757]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T01:28:26.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:25 smithi064 ceph-mon[18515]: from='osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T01:28:26.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:25 smithi064 ceph-mon[18515]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T01:28:27.054 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:26 smithi161 ceph-mon[21757]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:27.055 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:26 smithi161 ceph-mon[21757]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-19T01:28:27.055 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:26 smithi161 ceph-mon[21757]: from='osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:28:27.055 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:26 smithi161 ceph-mon[21757]: osdmap e46: 8 total, 7 up, 8 in 2024-08-19T01:28:27.055 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:26 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:27.055 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:26 smithi161 ceph-mon[21757]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:28:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:26 smithi123 ceph-mon[21829]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:26 smithi123 ceph-mon[21829]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-19T01:28:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:26 smithi123 ceph-mon[21829]: from='osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:28:27.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:26 smithi123 ceph-mon[21829]: osdmap e46: 8 total, 7 up, 8 in 2024-08-19T01:28:27.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:26 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:27.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:26 smithi123 ceph-mon[21829]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:28:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:26 smithi064 ceph-mon[18515]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:27.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:26 smithi064 ceph-mon[18515]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-19T01:28:27.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:26 smithi064 ceph-mon[18515]: from='osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:28:27.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:26 smithi064 ceph-mon[18515]: osdmap e46: 8 total, 7 up, 8 in 2024-08-19T01:28:27.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:26 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:27.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:26 smithi064 ceph-mon[18515]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:28:27.577 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:28:27.251+0000 7f6ad5c28640 -1 osd.7 0 waiting for initial osdmap 2024-08-19T01:28:27.577 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:28:27.260+0000 7f6acda48640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:28:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:28:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: osdmap e47: 8 total, 7 up, 8 in 2024-08-19T01:28:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: from='osd.7 ' entity='osd.7' 2024-08-19T01:28:27.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] boot 2024-08-19T01:28:27.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: osdmap e48: 8 total, 8 up, 8 in 2024-08-19T01:28:27.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:27 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:28:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: osdmap e47: 8 total, 7 up, 8 in 2024-08-19T01:28:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: from='osd.7 ' entity='osd.7' 2024-08-19T01:28:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] boot 2024-08-19T01:28:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: osdmap e48: 8 total, 8 up, 8 in 2024-08-19T01:28:28.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:27 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: osdmap e47: 8 total, 7 up, 8 in 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: from='osd.7 ' entity='osd.7' 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: osd.7 [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] boot 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: osdmap e48: 8 total, 8 up, 8 in 2024-08-19T01:28:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:27 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:28:28.408 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:28:28.433 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_2 2024-08-19T01:28:28.691 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi123 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi123 to 5696M 2024-08-19T01:28:28.972 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:28.973 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:28.973 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:28.973 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:28.973 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:28 smithi161 ceph-mon[21757]: osdmap e49: 8 total, 8 up, 8 in 2024-08-19T01:28:29.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:28:29.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:28:29.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi123 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi123 to 5696M 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:28 smithi064 ceph-mon[18515]: osdmap e49: 8 total, 8 up, 8 in 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi123 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:29.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:29.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi123 to 5696M 2024-08-19T01:28:29.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:29.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:29.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:29.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:28 smithi123 ceph-mon[21829]: osdmap e49: 8 total, 8 up, 8 in 2024-08-19T01:28:29.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 -- 172.21.15.161:0/2881544943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 msgr2=0x7f51f4109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:29.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 --2- 172.21.15.161:0/2881544943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f4109d90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009ca0 tx=0x7f51ec02a8c0 comp rx=0 tx=0).stop 2024-08-19T01:28:29.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 -- 172.21.15.161:0/2881544943 shutdown_connections 2024-08-19T01:28:29.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 --2- 172.21.15.161:0/2881544943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f4109d90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:29.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 --2- 172.21.15.161:0/2881544943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f4102b40 0x7f51f4102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:29.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 --2- 172.21.15.161:0/2881544943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51f4101940 0x7f51f4101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 -- 172.21.15.161:0/2881544943 >> 172.21.15.161:0/2881544943 conn(0x7f51f40fd0f0 msgr2=0x7f51f40ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 -- 172.21.15.161:0/2881544943 shutdown_connections 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 -- 172.21.15.161:0/2881544943 wait complete. 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 Processor -- start 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.193+0000 7f51fc65e640 1 -- start start 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fc65e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51f4101940 0x7f51f4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fc65e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f4102b40 0x7f51f41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fc65e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fc65e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410d5a0 con 0x7f51f4101940 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fc65e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410d420 con 0x7f51f4102b40 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fc65e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410d720 con 0x7f51f4103500 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f419f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53546/0 (socket says 172.21.15.161:53546) 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 -- 172.21.15.161:0/1066245276 learned_addr learned my addr 172.21.15.161:0/1066245276 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51f9bf9640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f4102b40 0x7f51f41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fa3fa640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51f4101940 0x7f51f4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 -- 172.21.15.161:0/1066245276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f4102b40 msgr2=0x7f51f41980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f4102b40 0x7f51f41980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 -- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51f4101940 msgr2=0x7f51f4197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51f4101940 0x7f51f4197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 -- 172.21.15.161:0/1066245276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f419f7c0 con 0x7f51f4103500 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51f9bf9640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f4102b40 0x7f51f41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fa3fa640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51f4101940 0x7f51f4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:28:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51fabfb640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f419f150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f51ec005e30 tx=0x7f51ec031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.194+0000 7f51e77fe640 1 -- 172.21.15.161:0/1066245276 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec003c30 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.195+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f419fa50 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.195+0000 7f51e77fe640 1 -- 172.21.15.161:0/1066245276 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51ec003dd0 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.195+0000 7f51e77fe640 1 -- 172.21.15.161:0/1066245276 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec02fde0 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.195+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f419ffe0 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.196+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f4108d10 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.197+0000 7f51e77fe640 1 -- 172.21.15.161:0/1066245276 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f51ec004050 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.197+0000 7f51e77fe640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f51e00a2c30 0x7f51e00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.198+0000 7f51e77fe640 1 -- 172.21.15.161:0/1066245276 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5426+0+0 (secure 0 0 0) 0x7f51ec0ef7a0 con 0x7f51f4103500 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.198+0000 7f51fa3fa640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f51e00a2c30 0x7f51e00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.198+0000 7f51fa3fa640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f51e00a2c30 0x7f51e00a50f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f51f0005d00 tx=0x7f51f0005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:28:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.202+0000 7f51e77fe640 1 -- 172.21.15.161:0/1066245276 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f51ec0bd240 con 0x7f51f4103500 2024-08-19T01:28:29.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:29.384+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f51f4062360 con 0x7f51e00a2c30 2024-08-19T01:28:29.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:29 smithi161 ceph-mon[21757]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:29.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:29.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:29.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:29 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:29.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:29 smithi161 ceph-mon[21757]: osdmap e50: 8 total, 8 up, 8 in 2024-08-19T01:28:30.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:29 smithi064 ceph-mon[18515]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:30.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:30.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:30.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:29 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:30.152 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:29 smithi064 ceph-mon[18515]: osdmap e50: 8 total, 8 up, 8 in 2024-08-19T01:28:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:29 smithi123 ceph-mon[21829]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:29 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:29 smithi123 ceph-mon[21829]: osdmap e50: 8 total, 8 up, 8 in 2024-08-19T01:28:31.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:31 smithi064 ceph-mon[18515]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:31.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:31 smithi064 ceph-mon[18515]: osdmap e51: 8 total, 8 up, 8 in 2024-08-19T01:28:31.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:31 smithi123 ceph-mon[21829]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:31.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:31 smithi123 ceph-mon[21829]: osdmap e51: 8 total, 8 up, 8 in 2024-08-19T01:28:31.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:31 smithi161 ceph-mon[21757]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:31.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:31 smithi161 ceph-mon[21757]: osdmap e51: 8 total, 8 up, 8 in 2024-08-19T01:28:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:32 smithi123 ceph-mon[21829]: osdmap e52: 8 total, 8 up, 8 in 2024-08-19T01:28:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:32 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/1690889045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]: dispatch 2024-08-19T01:28:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:32 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]: dispatch 2024-08-19T01:28:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:32 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]': finished 2024-08-19T01:28:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:32 smithi123 ceph-mon[21829]: osdmap e53: 9 total, 8 up, 9 in 2024-08-19T01:28:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:32 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:32 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/1713916511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:32 smithi161 ceph-mon[21757]: osdmap e52: 8 total, 8 up, 8 in 2024-08-19T01:28:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:32 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/1690889045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]: dispatch 2024-08-19T01:28:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:32 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]: dispatch 2024-08-19T01:28:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:32 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]': finished 2024-08-19T01:28:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:32 smithi161 ceph-mon[21757]: osdmap e53: 9 total, 8 up, 9 in 2024-08-19T01:28:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:32 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:32.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:32 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/1713916511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:32 smithi064 ceph-mon[18515]: osdmap e52: 8 total, 8 up, 8 in 2024-08-19T01:28:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:32 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/1690889045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]: dispatch 2024-08-19T01:28:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:32 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]: dispatch 2024-08-19T01:28:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:32 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5b64329-e576-4c16-87ae-2180430f9ca5"}]': finished 2024-08-19T01:28:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:32 smithi064 ceph-mon[18515]: osdmap e53: 9 total, 8 up, 9 in 2024-08-19T01:28:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:32 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:32.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:32 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/1713916511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:33.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:33 smithi123 ceph-mon[21829]: pgmap v161: 1 pgs: 1 remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:33.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:33 smithi161 ceph-mon[21757]: pgmap v161: 1 pgs: 1 remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:33.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:33 smithi064 ceph-mon[18515]: pgmap v161: 1 pgs: 1 remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:35.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:35 smithi161 ceph-mon[21757]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:35 smithi123 ceph-mon[21829]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:35.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:35 smithi064 ceph-mon[18515]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:37.822 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:37 smithi161 ceph-mon[21757]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:37.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:37 smithi123 ceph-mon[21829]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:37 smithi064 ceph-mon[18515]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:39.828 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:39 smithi161 ceph-mon[21757]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:39.828 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-08-19T01:28:39.828 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:39.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:39 smithi123 ceph-mon[21829]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:39.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-08-19T01:28:39.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:39.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:39 smithi064 ceph-mon[18515]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:39.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-08-19T01:28:39.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:40.835 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:40 smithi161 ceph-mon[21757]: Deploying daemon osd.8 on smithi161 2024-08-19T01:28:40.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:40 smithi064 ceph-mon[18515]: Deploying daemon osd.8 on smithi161 2024-08-19T01:28:40.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:40 smithi123 ceph-mon[21829]: Deploying daemon osd.8 on smithi161 2024-08-19T01:28:41.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:41 smithi161 ceph-mon[21757]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:41.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:41 smithi064 ceph-mon[18515]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:41.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:41 smithi123 ceph-mon[21829]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:43.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:43 smithi123 ceph-mon[21829]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:43.918 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:43 smithi161 ceph-mon[21757]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:43.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:43 smithi064 ceph-mon[18515]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:44.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:44.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:44.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:44.908 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:44.908 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:44.908 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:44.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:44.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:44.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:45.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:45 smithi123 ceph-mon[21829]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:45.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:45 smithi161 ceph-mon[21757]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:45.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:45 smithi064 ceph-mon[18515]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:47.811 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:47 smithi161 ceph-mon[21757]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:47.811 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.811 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.811 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:47.811 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:47.811 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.850+0000 7f51e77fe640 1 -- 172.21.15.161:0/1066245276 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f51f4062360 con 0x7f51e00a2c30 2024-08-19T01:28:47.853 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 8 on host 'smithi161' 2024-08-19T01:28:47.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f51e00a2c30 msgr2=0x7f51e00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:47.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f51e00a2c30 0x7f51e00a50f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f51f0005d00 tx=0x7f51f0005c90 comp rx=0 tx=0).stop 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 msgr2=0x7f51f419f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f419f150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f51ec005e30 tx=0x7f51ec031ee0 comp rx=0 tx=0).stop 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 shutdown_connections 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f51e00a2c30 0x7f51e00a50f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4103500 0x7f51f419f150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f4102b40 0x7f51f41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 --2- 172.21.15.161:0/1066245276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51f4101940 0x7f51f4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 >> 172.21.15.161:0/1066245276 conn(0x7f51f40fd0f0 msgr2=0x7f51f4105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 shutdown_connections 2024-08-19T01:28:47.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:28:47.853+0000 7f51fc65e640 1 -- 172.21.15.161:0/1066245276 wait complete. 2024-08-19T01:28:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:47 smithi123 ceph-mon[21829]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:47 smithi064 ceph-mon[18515]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:47.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:47.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:47.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:47.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:48.551 DEBUG:teuthology.orchestra.run.smithi161:osd.8> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.8.service 2024-08-19T01:28:48.553 INFO:tasks.cephadm:Deploying osd.9 on smithi064 with /dev/vg_nvme/lv_1... 2024-08-19T01:28:48.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-19T01:28:48.826 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:48.827 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:48.827 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:48.827 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:48 smithi161 ceph-mon[21757]: from='osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T01:28:48.827 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:48 smithi161 ceph-mon[21757]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T01:28:48.827 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:28:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:48 smithi123 ceph-mon[21829]: from='osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T01:28:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:48 smithi123 ceph-mon[21829]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T01:28:48.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:28:48.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:48.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:48.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:48 smithi064 ceph-mon[18515]: from='osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T01:28:48.950 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:48 smithi064 ceph-mon[18515]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T01:28:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-mon[21757]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-mon[21757]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-19T01:28:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-mon[21757]: osdmap e54: 9 total, 8 up, 9 in 2024-08-19T01:28:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-mon[21757]: from='osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:28:49.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-mon[21757]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:28:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:49 smithi123 ceph-mon[21829]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:49 smithi123 ceph-mon[21829]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-19T01:28:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:49 smithi123 ceph-mon[21829]: osdmap e54: 9 total, 8 up, 9 in 2024-08-19T01:28:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:49 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:49 smithi123 ceph-mon[21829]: from='osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:28:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:49 smithi123 ceph-mon[21829]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:28:49.975 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:49 smithi064 ceph-mon[18515]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:49.975 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:49 smithi064 ceph-mon[18515]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-19T01:28:49.976 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:49 smithi064 ceph-mon[18515]: osdmap e54: 9 total, 8 up, 9 in 2024-08-19T01:28:49.976 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:49 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:49.976 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:49 smithi064 ceph-mon[18515]: from='osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:28:49.976 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:49 smithi064 ceph-mon[18515]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:28:50.039 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:28:49.855+0000 7fde8c90d640 -1 osd.8 0 waiting for initial osdmap 2024-08-19T01:28:50.039 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:28:49 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:28:49.866+0000 7fde88735640 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:28:50.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:50 smithi064 ceph-mon[18515]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:50.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:50 smithi064 ceph-mon[18515]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:28:50.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:50 smithi064 ceph-mon[18515]: osdmap e55: 9 total, 8 up, 9 in 2024-08-19T01:28:50.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:50.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:51.138 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:50 smithi161 ceph-mon[21757]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:51.138 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:50 smithi161 ceph-mon[21757]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:28:51.138 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:50 smithi161 ceph-mon[21757]: osdmap e55: 9 total, 8 up, 9 in 2024-08-19T01:28:51.138 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:51.138 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:51.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:50 smithi123 ceph-mon[21829]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-08-19T01:28:51.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:50 smithi123 ceph-mon[21829]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:28:51.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:50 smithi123 ceph-mon[21829]: osdmap e55: 9 total, 8 up, 9 in 2024-08-19T01:28:51.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:51.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:51.609 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:28:51.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_1 2024-08-19T01:28:51.903 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:28:52.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:28:52.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:28:52.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] boot 2024-08-19T01:28:52.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: osdmap e56: 9 total, 9 up, 9 in 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi161 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi161 to 5696M 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:52.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] boot 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: osdmap e56: 9 total, 9 up, 9 in 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi161 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi161 to 5696M 2024-08-19T01:28:52.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:52.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:52.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: osd.8 [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] boot 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: osdmap e56: 9 total, 9 up, 9 in 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi161 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi161 to 5696M 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:28:52.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:28:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 -- 172.21.15.64:0/4080728939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd304107630 msgr2=0x7fd304109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 --2- 172.21.15.64:0/4080728939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd304107630 0x7fd304109a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009ca0 tx=0x7fd2fc02a8c0 comp rx=0 tx=0).stop 2024-08-19T01:28:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 -- 172.21.15.64:0/4080728939 shutdown_connections 2024-08-19T01:28:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 --2- 172.21.15.64:0/4080728939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd304107630 0x7fd304109a80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 --2- 172.21.15.64:0/4080728939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd304104d00 0x7fd3041070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 --2- 172.21.15.64:0/4080728939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3041023d0 0x7fd3041047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 -- 172.21.15.64:0/4080728939 >> 172.21.15.64:0/4080728939 conn(0x7fd3040fbfd0 msgr2=0x7fd3040fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 -- 172.21.15.64:0/4080728939 shutdown_connections 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.394+0000 7fd30a785640 1 -- 172.21.15.64:0/4080728939 wait complete. 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 Processor -- start 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 -- start start 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3041023d0 0x7fd304101d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd304104d00 0x7fd304100380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd304107630 0x7fd3041008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd30410d5d0 con 0x7fd3041023d0 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd30410d450 con 0x7fd304104d00 2024-08-19T01:28:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd30a785640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd30410d750 con 0x7fd304107630 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3041023d0 0x7fd304101d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3041023d0 0x7fd304101d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33974/0 (socket says 172.21.15.64:33974) 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 -- 172.21.15.64:0/141123096 learned_addr learned my addr 172.21.15.64:0/141123096 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd3037fe640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd304104d00 0x7fd304100380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 -- 172.21.15.64:0/141123096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd304107630 msgr2=0x7fd3041008c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd304107630 0x7fd3041008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 -- 172.21.15.64:0/141123096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd304104d00 msgr2=0x7fd304100380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd304104d00 0x7fd304100380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.395+0000 7fd303fff640 1 -- 172.21.15.64:0/141123096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd304101180 con 0x7fd3041023d0 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.396+0000 7fd303fff640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3041023d0 0x7fd304101d30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c920 tx=0x7fd2f400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.396+0000 7fd3017fa640 1 -- 172.21.15.64:0/141123096 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4007c70 con 0x7fd3041023d0 2024-08-19T01:28:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.396+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3041a1d10 con 0x7fd3041023d0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.396+0000 7fd3017fa640 1 -- 172.21.15.64:0/141123096 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2f4007e10 con 0x7fd3041023d0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.396+0000 7fd3017fa640 1 -- 172.21.15.64:0/141123096 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4018890 con 0x7fd3041023d0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.396+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3041a21c0 con 0x7fd3041023d0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.397+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3040fdac0 con 0x7fd3041023d0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.398+0000 7fd3017fa640 1 -- 172.21.15.64:0/141123096 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd2f4018a50 con 0x7fd3041023d0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.399+0000 7fd3017fa640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd2ec0a2b80 0x7fd2ec0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.399+0000 7fd3037fe640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd2ec0a2b80 0x7fd2ec0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.399+0000 7fd3017fa640 1 -- 172.21.15.64:0/141123096 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5858+0+0 (secure 0 0 0) 0x7fd2f40ce8d0 con 0x7fd3041023d0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.399+0000 7fd3037fe640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd2ec0a2b80 0x7fd2ec0a5040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005fd0 tx=0x7fd2f8009470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:28:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.407+0000 7fd3017fa640 1 -- 172.21.15.64:0/141123096 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd2f4014030 con 0x7fd3041023d0 2024-08-19T01:28:52.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:28:52.594+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd304062360 con 0x7fd2ec0a2b80 2024-08-19T01:28:53.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:52 smithi123 ceph-mon[21829]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:53.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:52 smithi123 ceph-mon[21829]: osdmap e57: 9 total, 9 up, 9 in 2024-08-19T01:28:53.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:52 smithi123 ceph-mon[21829]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:53.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:52 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:53.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:52 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:53.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:52 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:53.169 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:52 smithi064 ceph-mon[18515]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:53.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:52 smithi064 ceph-mon[18515]: osdmap e57: 9 total, 9 up, 9 in 2024-08-19T01:28:53.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:52 smithi064 ceph-mon[18515]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:53.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:52 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:53.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:52 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:53.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:52 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:53.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:52 smithi161 ceph-mon[21757]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:53.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:52 smithi161 ceph-mon[21757]: osdmap e57: 9 total, 9 up, 9 in 2024-08-19T01:28:53.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:52 smithi161 ceph-mon[21757]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:28:53.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:52 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:28:53.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:52 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:28:53.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:52 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:28:54.142 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:53 smithi064 ceph-mon[18515]: osdmap e58: 9 total, 9 up, 9 in 2024-08-19T01:28:54.143 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:53 smithi064 ceph-mon[18515]: osdmap e59: 9 total, 9 up, 9 in 2024-08-19T01:28:54.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:53 smithi123 ceph-mon[21829]: osdmap e58: 9 total, 9 up, 9 in 2024-08-19T01:28:54.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:53 smithi123 ceph-mon[21829]: osdmap e59: 9 total, 9 up, 9 in 2024-08-19T01:28:54.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:53 smithi161 ceph-mon[21757]: osdmap e58: 9 total, 9 up, 9 in 2024-08-19T01:28:54.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:53 smithi161 ceph-mon[21757]: osdmap e59: 9 total, 9 up, 9 in 2024-08-19T01:28:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:55 smithi064 ceph-mon[18515]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:55 smithi064 ceph-mon[18515]: osdmap e60: 9 total, 9 up, 9 in 2024-08-19T01:28:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:55 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1948000899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50795cb0-f711-4f52-b8fa-e9583c165cae"}]: dispatch 2024-08-19T01:28:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:55 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1948000899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50795cb0-f711-4f52-b8fa-e9583c165cae"}]': finished 2024-08-19T01:28:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:55 smithi064 ceph-mon[18515]: osdmap e61: 10 total, 9 up, 10 in 2024-08-19T01:28:55.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:55 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:28:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:55 smithi123 ceph-mon[21829]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:55 smithi123 ceph-mon[21829]: osdmap e60: 9 total, 9 up, 9 in 2024-08-19T01:28:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:55 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1948000899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50795cb0-f711-4f52-b8fa-e9583c165cae"}]: dispatch 2024-08-19T01:28:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:55 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1948000899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50795cb0-f711-4f52-b8fa-e9583c165cae"}]': finished 2024-08-19T01:28:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:55 smithi123 ceph-mon[21829]: osdmap e61: 10 total, 9 up, 10 in 2024-08-19T01:28:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:55 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:28:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:55 smithi161 ceph-mon[21757]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:55 smithi161 ceph-mon[21757]: osdmap e60: 9 total, 9 up, 9 in 2024-08-19T01:28:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:55 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1948000899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50795cb0-f711-4f52-b8fa-e9583c165cae"}]: dispatch 2024-08-19T01:28:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:55 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1948000899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50795cb0-f711-4f52-b8fa-e9583c165cae"}]': finished 2024-08-19T01:28:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:55 smithi161 ceph-mon[21757]: osdmap e61: 10 total, 9 up, 10 in 2024-08-19T01:28:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:55 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:28:56.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:56 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1339299841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:56.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:56 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1339299841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:56.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:56 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1339299841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:28:57.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:57 smithi123 ceph-mon[21829]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:57 smithi161 ceph-mon[21757]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:57.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:57 smithi064 ceph-mon[18515]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:59.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:28:59 smithi123 ceph-mon[21829]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:28:59 smithi161 ceph-mon[21757]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:28:59.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:28:59 smithi064 ceph-mon[18515]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:29:01.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:01 smithi123 ceph-mon[21829]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-19T01:29:01.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:01 smithi161 ceph-mon[21757]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-19T01:29:01.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:01 smithi064 ceph-mon[18515]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-19T01:29:02.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-08-19T01:29:02.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:02 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:02.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:02 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-08-19T01:29:02.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:02 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:02 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-08-19T01:29:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:02 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:03.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:03 smithi123 ceph-mon[21829]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T01:29:03.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:03 smithi123 ceph-mon[21829]: Deploying daemon osd.9 on smithi064 2024-08-19T01:29:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:03 smithi161 ceph-mon[21757]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T01:29:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:03 smithi161 ceph-mon[21757]: Deploying daemon osd.9 on smithi064 2024-08-19T01:29:03.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:03 smithi064 ceph-mon[18515]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T01:29:03.970 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:03 smithi064 ceph-mon[18515]: Deploying daemon osd.9 on smithi064 2024-08-19T01:29:05.915 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:05 smithi064 ceph-mon[18515]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-08-19T01:29:05.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:05 smithi123 ceph-mon[21829]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-08-19T01:29:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:05 smithi161 ceph-mon[21757]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-08-19T01:29:07.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:07 smithi123 ceph-mon[21829]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-19T01:29:07.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:07 smithi161 ceph-mon[21757]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-19T01:29:07.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:07 smithi064 ceph-mon[18515]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-19T01:29:08.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:08 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:08.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:08 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:08.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:08 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:08.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:08 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:08.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:08 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:08.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:08 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:08.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:08 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:08.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:08 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:08.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:08 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:09.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:09 smithi064 ceph-mon[18515]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:09.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:09 smithi123 ceph-mon[21829]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:09 smithi161 ceph-mon[21757]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:11.427 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 9 on host 'smithi064' 2024-08-19T01:29:11.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.425+0000 7fd3017fa640 1 -- 172.21.15.64:0/141123096 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd304062360 con 0x7fd2ec0a2b80 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd2ec0a2b80 msgr2=0x7fd2ec0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd2ec0a2b80 0x7fd2ec0a5040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005fd0 tx=0x7fd2f8009470 comp rx=0 tx=0).stop 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3041023d0 msgr2=0x7fd304101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3041023d0 0x7fd304101d30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c920 tx=0x7fd2f400cde0 comp rx=0 tx=0).stop 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 shutdown_connections 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd2ec0a2b80 0x7fd2ec0a5040 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd304107630 0x7fd3041008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd304104d00 0x7fd304100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:11.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 --2- 172.21.15.64:0/141123096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3041023d0 0x7fd304101d30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:11.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 >> 172.21.15.64:0/141123096 conn(0x7fd3040fbfd0 msgr2=0x7fd304102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:29:11.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 shutdown_connections 2024-08-19T01:29:11.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:11.427+0000 7fd30a785640 1 -- 172.21.15.64:0/141123096 wait complete. 2024-08-19T01:29:11.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:11 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:11 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:11.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:11.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:11 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:12.129 DEBUG:teuthology.orchestra.run.smithi064:osd.9> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.9.service 2024-08-19T01:29:12.132 INFO:tasks.cephadm:Deploying osd.10 on smithi123 with /dev/vg_nvme/lv_1... 2024-08-19T01:29:12.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-19T01:29:12.394 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:29:12.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:12 smithi064 ceph-mon[18515]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T01:29:12.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:12 smithi123 ceph-mon[21829]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T01:29:12.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:12 smithi161 ceph-mon[21757]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T01:29:13.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:13 smithi064 ceph-mon[18515]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:29:13.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:13 smithi064 ceph-mon[18515]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-19T01:29:13.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:13 smithi064 ceph-mon[18515]: osdmap e62: 10 total, 9 up, 10 in 2024-08-19T01:29:13.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:13 smithi064 ceph-mon[18515]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:29:13.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:13.907 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:13 smithi123 ceph-mon[21829]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:29:13.907 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:13 smithi123 ceph-mon[21829]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-19T01:29:13.907 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:13 smithi123 ceph-mon[21829]: osdmap e62: 10 total, 9 up, 10 in 2024-08-19T01:29:13.907 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:13 smithi123 ceph-mon[21829]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:29:13.907 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:13 smithi161 ceph-mon[21757]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:29:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:13 smithi161 ceph-mon[21757]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-19T01:29:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:13 smithi161 ceph-mon[21757]: osdmap e62: 10 total, 9 up, 10 in 2024-08-19T01:29:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:13 smithi161 ceph-mon[21757]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-08-19T01:29:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:14.420 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:29:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:29:14.040+0000 7fae7c72a640 -1 osd.9 0 waiting for initial osdmap 2024-08-19T01:29:14.420 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:29:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:29:14.051+0000 7fae7454a640 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:29:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:14 smithi064 ceph-mon[18515]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:29:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:14 smithi064 ceph-mon[18515]: osdmap e63: 10 total, 9 up, 10 in 2024-08-19T01:29:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:14 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:14.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:14 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:14.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:14 smithi064 ceph-mon[18515]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' 2024-08-19T01:29:14.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:14 smithi123 ceph-mon[21829]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:29:14.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:14 smithi123 ceph-mon[21829]: osdmap e63: 10 total, 9 up, 10 in 2024-08-19T01:29:14.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:14 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:14.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:14 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:14.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:14 smithi123 ceph-mon[21829]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' 2024-08-19T01:29:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:14 smithi161 ceph-mon[21757]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-08-19T01:29:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:14 smithi161 ceph-mon[21757]: osdmap e63: 10 total, 9 up, 10 in 2024-08-19T01:29:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:14 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:14 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:14 smithi161 ceph-mon[21757]: from='osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660]' entity='osd.9' 2024-08-19T01:29:15.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-08-19T01:29:15.136 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi123:vg_nvme/lv_1 2024-08-19T01:29:15.392 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] boot 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: osdmap e64: 10 total, 10 up, 10 in 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:15.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:15 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:29:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:29:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:29:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] boot 2024-08-19T01:29:15.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: osdmap e64: 10 total, 10 up, 10 in 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:15.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:15 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 -- 172.21.15.123:0/4024286575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c102400 msgr2=0x7f5e9c108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/4024286575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c102400 0x7f5e9c108c90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009ca0 tx=0x7f5e9402a8c0 comp rx=0 tx=0).stop 2024-08-19T01:29:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 -- 172.21.15.123:0/4024286575 shutdown_connections 2024-08-19T01:29:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/4024286575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c102400 0x7f5e9c108c90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/4024286575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c101a40 0x7f5e9c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/4024286575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e9c100840 0x7f5e9c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 -- 172.21.15.123:0/4024286575 >> 172.21.15.123:0/4024286575 conn(0x7f5e9c0fbfd0 msgr2=0x7f5e9c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 -- 172.21.15.123:0/4024286575 shutdown_connections 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 -- 172.21.15.123:0/4024286575 wait complete. 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.882+0000 7f5ea2f96640 1 Processor -- start 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea2f96640 1 -- start start 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea2f96640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c100840 0x7f5e9c199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea2f96640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c101a40 0x7f5e9c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea2f96640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e9c102400 0x7f5e9c1a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea2f96640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10c4c0 con 0x7f5e9c102400 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea2f96640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10c340 con 0x7f5e9c100840 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea2f96640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10c640 con 0x7f5e9c101a40 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c100840 0x7f5e9c199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c100840 0x7f5e9c199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35136/0 (socket says 172.21.15.123:35136) 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c101a40 0x7f5e9c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 -- 172.21.15.123:0/1710176967 learned_addr learned my addr 172.21.15.123:0/1710176967 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea1533640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e9c102400 0x7f5e9c1a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 -- 172.21.15.123:0/1710176967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c101a40 msgr2=0x7f5e9c19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c101a40 0x7f5e9c19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 -- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e9c102400 msgr2=0x7f5e9c1a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e9c102400 0x7f5e9c1a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 -- 172.21.15.123:0/1710176967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9c1a1a40 con 0x7f5e9c100840 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.883+0000 7f5ea0d32640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c100840 0x7f5e9c199e10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c970 tx=0x7f5e9000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.884+0000 7f5e99ffb640 1 -- 172.21.15.123:0/1710176967 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e90004220 con 0x7f5e9c100840 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.884+0000 7f5e99ffb640 1 -- 172.21.15.123:0/1710176967 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e900043c0 con 0x7f5e9c100840 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.884+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9c1a1d30 con 0x7f5e9c100840 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.884+0000 7f5e99ffb640 1 -- 172.21.15.123:0/1710176967 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9000f450 con 0x7f5e9c100840 2024-08-19T01:29:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.884+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9c1a21c0 con 0x7f5e9c100840 2024-08-19T01:29:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.885+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e9c1a24b0 con 0x7f5e9c100840 2024-08-19T01:29:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.886+0000 7f5e99ffb640 1 -- 172.21.15.123:0/1710176967 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5e900079b0 con 0x7f5e9c100840 2024-08-19T01:29:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.887+0000 7f5e99ffb640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5e8c0a2bb0 0x7f5e8c0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.887+0000 7f5e99ffb640 1 -- 172.21.15.123:0/1710176967 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6261+0+0 (secure 0 0 0) 0x7f5e900ce4d0 con 0x7f5e9c100840 2024-08-19T01:29:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.887+0000 7f5e9bfff640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5e8c0a2bb0 0x7f5e8c0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.887+0000 7f5e9bfff640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5e8c0a2bb0 0x7f5e8c0a5070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c19b330 tx=0x7f5e8800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:29:15.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:15.891+0000 7f5e99ffb640 1 -- 172.21.15.123:0/1710176967 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f5e90014030 con 0x7f5e9c100840 2024-08-19T01:29:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:29:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:29:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-08-19T01:29:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: osd.9 [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] boot 2024-08-19T01:29:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: osdmap e64: 10 total, 10 up, 10 in 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:15.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:15 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:16.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:16.065+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5e9c1a2680 con 0x7f5e8c0a2bb0 2024-08-19T01:29:16.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:16 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi064 2024-08-19T01:29:16.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:16 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi064 to 4272M 2024-08-19T01:29:16.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:29:16.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:29:16.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:16.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:16 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi064 2024-08-19T01:29:16.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:16 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi064 to 4272M 2024-08-19T01:29:16.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:29:16.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:29:16.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:16.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:16 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi064 2024-08-19T01:29:16.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:16 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi064 to 4272M 2024-08-19T01:29:16.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:29:16.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:29:16.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:17.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:17 smithi123 ceph-mon[21829]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:17.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:17 smithi123 ceph-mon[21829]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:29:17.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:17 smithi123 ceph-mon[21829]: osdmap e65: 10 total, 10 up, 10 in 2024-08-19T01:29:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:17 smithi161 ceph-mon[21757]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:17 smithi161 ceph-mon[21757]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:29:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:17 smithi161 ceph-mon[21757]: osdmap e65: 10 total, 10 up, 10 in 2024-08-19T01:29:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:17 smithi064 ceph-mon[18515]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:17 smithi064 ceph-mon[18515]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:29:17.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:17 smithi064 ceph-mon[18515]: osdmap e65: 10 total, 10 up, 10 in 2024-08-19T01:29:18.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:18 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/4000535860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]: dispatch 2024-08-19T01:29:18.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:18 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]: dispatch 2024-08-19T01:29:18.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:18 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]': finished 2024-08-19T01:29:18.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:18 smithi123 ceph-mon[21829]: osdmap e66: 11 total, 10 up, 11 in 2024-08-19T01:29:18.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:18 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:18 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/4000535860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]: dispatch 2024-08-19T01:29:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:18 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]: dispatch 2024-08-19T01:29:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:18 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]': finished 2024-08-19T01:29:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:18 smithi161 ceph-mon[21757]: osdmap e66: 11 total, 10 up, 11 in 2024-08-19T01:29:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:18 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:18 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/4000535860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]: dispatch 2024-08-19T01:29:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:18 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]: dispatch 2024-08-19T01:29:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:18 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe117900-4a44-493a-b389-37c19ce9356a"}]': finished 2024-08-19T01:29:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:18 smithi064 ceph-mon[18515]: osdmap e66: 11 total, 10 up, 11 in 2024-08-19T01:29:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:18 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:19 smithi123 ceph-mon[21829]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:19 smithi123 ceph-mon[21829]: from='client.? 172.21.15.123:0/2067521737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:29:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:19 smithi161 ceph-mon[21757]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:19 smithi161 ceph-mon[21757]: from='client.? 172.21.15.123:0/2067521737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:29:19.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:19 smithi064 ceph-mon[18515]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:19.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:19 smithi064 ceph-mon[18515]: from='client.? 172.21.15.123:0/2067521737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:29:20.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:20 smithi123 ceph-mon[21829]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:20.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:20 smithi161 ceph-mon[21757]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:20.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:20 smithi064 ceph-mon[18515]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:22.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:22 smithi123 ceph-mon[21829]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:22.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:22 smithi064 ceph-mon[18515]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:23.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:22 smithi161 ceph-mon[21757]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:24.955 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:24 smithi123 ceph-mon[21829]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:24.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:24 smithi064 ceph-mon[18515]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:25.001 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:24 smithi161 ceph-mon[21757]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:26.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:26 smithi123 ceph-mon[21829]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:26.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:26 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-08-19T01:29:26.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:26 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:26.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:26 smithi123 ceph-mon[21829]: Deploying daemon osd.10 on smithi123 2024-08-19T01:29:26.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:26 smithi064 ceph-mon[18515]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:26.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:26 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-08-19T01:29:26.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:26 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:26.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:26 smithi064 ceph-mon[18515]: Deploying daemon osd.10 on smithi123 2024-08-19T01:29:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:26 smithi161 ceph-mon[21757]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:26 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-08-19T01:29:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:26 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:26 smithi161 ceph-mon[21757]: Deploying daemon osd.10 on smithi123 2024-08-19T01:29:28.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:28 smithi123 ceph-mon[21829]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:28.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:28 smithi064 ceph-mon[18515]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:28 smithi161 ceph-mon[21757]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:30.909 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:30 smithi123 ceph-mon[21829]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:30.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:30 smithi064 ceph-mon[18515]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:30 smithi161 ceph-mon[21757]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:31.900 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:31.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:31.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:33.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:33 smithi123 ceph-mon[21829]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:33 smithi161 ceph-mon[21757]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:33.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:33 smithi064 ceph-mon[18515]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:34.690 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 10 on host 'smithi123' 2024-08-19T01:29:34.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.688+0000 7f5e99ffb640 1 -- 172.21.15.123:0/1710176967 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f5e9c1a2680 con 0x7f5e8c0a2bb0 2024-08-19T01:29:34.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.691+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5e8c0a2bb0 msgr2=0x7f5e8c0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:34.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.691+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5e8c0a2bb0 0x7f5e8c0a5070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c19b330 tx=0x7f5e8800b390 comp rx=0 tx=0).stop 2024-08-19T01:29:34.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.691+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c100840 msgr2=0x7f5e9c199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:34.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.691+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c100840 0x7f5e9c199e10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c970 tx=0x7f5e9000ce30 comp rx=0 tx=0).stop 2024-08-19T01:29:34.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 shutdown_connections 2024-08-19T01:29:34.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5e8c0a2bb0 0x7f5e8c0a5070 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:34.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e9c102400 0x7f5e9c1a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:34.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c101a40 0x7f5e9c19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:34.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 --2- 172.21.15.123:0/1710176967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c100840 0x7f5e9c199e10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:34.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 >> 172.21.15.123:0/1710176967 conn(0x7f5e9c0fbfd0 msgr2=0x7f5e9c06a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:29:34.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 shutdown_connections 2024-08-19T01:29:34.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-08-19T01:29:34.692+0000 7f5ea2f96640 1 -- 172.21.15.123:0/1710176967 wait complete. 2024-08-19T01:29:35.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:35 smithi123 ceph-mon[21829]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:35.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:35.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:35.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:35 smithi123 ceph-mon[21829]: from='osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T01:29:35.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:35 smithi123 ceph-mon[21829]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T01:29:35.358 DEBUG:teuthology.orchestra.run.smithi123:osd.10> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.10.service 2024-08-19T01:29:35.360 INFO:tasks.cephadm:Deploying osd.11 on smithi161 with /dev/vg_nvme/lv_1... 2024-08-19T01:29:35.360 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-19T01:29:35.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:35 smithi161 ceph-mon[21757]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:35.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:35.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:35.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:35 smithi161 ceph-mon[21757]: from='osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T01:29:35.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:35 smithi161 ceph-mon[21757]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T01:29:35.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:35 smithi064 ceph-mon[18515]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:35.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:35.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:35.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:35 smithi064 ceph-mon[18515]: from='osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T01:29:35.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:35 smithi064 ceph-mon[18515]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T01:29:35.623 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:29:36.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:36.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:36.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:36 smithi123 ceph-mon[21829]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T01:29:36.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:36 smithi123 ceph-mon[21829]: osdmap e67: 11 total, 10 up, 11 in 2024-08-19T01:29:36.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:36 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:36.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:36 smithi123 ceph-mon[21829]: from='osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:29:36.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:36 smithi123 ceph-mon[21829]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:29:36.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:36.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:36.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:36 smithi161 ceph-mon[21757]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T01:29:36.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:36 smithi161 ceph-mon[21757]: osdmap e67: 11 total, 10 up, 11 in 2024-08-19T01:29:36.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:36 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:36.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:36 smithi161 ceph-mon[21757]: from='osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:29:36.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:36 smithi161 ceph-mon[21757]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:29:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:36 smithi064 ceph-mon[18515]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T01:29:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:36 smithi064 ceph-mon[18515]: osdmap e67: 11 total, 10 up, 11 in 2024-08-19T01:29:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:36 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:36 smithi064 ceph-mon[18515]: from='osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:29:36.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:36 smithi064 ceph-mon[18515]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-08-19T01:29:37.393 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:29:37.116+0000 7fa90bb65640 -1 osd.10 0 waiting for initial osdmap 2024-08-19T01:29:37.393 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:29:37.126+0000 7fa903985640 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:29:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-mon[21829]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-mon[21829]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:29:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-mon[21829]: osdmap e68: 11 total, 10 up, 11 in 2024-08-19T01:29:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:37 smithi123 ceph-mon[21829]: from='osd.10 ' entity='osd.10' 2024-08-19T01:29:37.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:37 smithi161 ceph-mon[21757]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:37.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:37 smithi161 ceph-mon[21757]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:29:37.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:37 smithi161 ceph-mon[21757]: osdmap e68: 11 total, 10 up, 11 in 2024-08-19T01:29:37.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:37.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:37.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:37 smithi161 ceph-mon[21757]: from='osd.10 ' entity='osd.10' 2024-08-19T01:29:37.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:37 smithi064 ceph-mon[18515]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:37.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:37 smithi064 ceph-mon[18515]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-08-19T01:29:37.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:37 smithi064 ceph-mon[18515]: osdmap e68: 11 total, 10 up, 11 in 2024-08-19T01:29:37.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:37.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:37.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:37 smithi064 ceph-mon[18515]: from='osd.10 ' entity='osd.10' 2024-08-19T01:29:38.155 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-08-19T01:29:38.181 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_1 2024-08-19T01:29:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:29:38.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:38 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:29:38.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:38 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:29:38.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:38 smithi161 ceph-mon[21757]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:38.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:38 smithi161 ceph-mon[21757]: osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] boot 2024-08-19T01:29:38.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:38 smithi161 ceph-mon[21757]: osdmap e69: 11 total, 11 up, 11 in 2024-08-19T01:29:38.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:38 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:38.840 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:38 smithi161 ceph-mon[21757]: osdmap e70: 11 total, 11 up, 11 in 2024-08-19T01:29:38.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:38 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:29:38.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:38 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:29:38.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:38 smithi123 ceph-mon[21829]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:38.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:38 smithi123 ceph-mon[21829]: osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] boot 2024-08-19T01:29:38.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:38 smithi123 ceph-mon[21829]: osdmap e69: 11 total, 11 up, 11 in 2024-08-19T01:29:38.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:38 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:38.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:38 smithi123 ceph-mon[21829]: osdmap e70: 11 total, 11 up, 11 in 2024-08-19T01:29:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- 172.21.15.161:0/507486631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 msgr2=0x7f3440101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 --2- 172.21.15.161:0/507486631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440101d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3430009900 tx=0x7f343002a410 comp rx=0 tx=0).stop 2024-08-19T01:29:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- 172.21.15.161:0/507486631 shutdown_connections 2024-08-19T01:29:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 --2- 172.21.15.161:0/507486631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34401034f0 0x7f3440109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 --2- 172.21.15.161:0/507486631 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3440102b30 0x7f3440102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 --2- 172.21.15.161:0/507486631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440101d30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- 172.21.15.161:0/507486631 >> 172.21.15.161:0/507486631 conn(0x7f34400fd0c0 msgr2=0x7f34400ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- 172.21.15.161:0/507486631 shutdown_connections 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- 172.21.15.161:0/507486631 wait complete. 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 Processor -- start 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- start start 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3440102b30 0x7f34401981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34401034f0 0x7f344019f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f344010d590 con 0x7f34401034f0 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f344010d410 con 0x7f3440102b30 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3446cb8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f344010d710 con 0x7f3440101930 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3444a54640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3444a54640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45848/0 (socket says 172.21.15.161:45848) 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.922+0000 7f3444a54640 1 -- 172.21.15.161:0/1702514314 learned_addr learned my addr 172.21.15.161:0/1702514314 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f343ffff640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3440102b30 0x7f34401981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3444a54640 1 -- 172.21.15.161:0/1702514314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3440102b30 msgr2=0x7f34401981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3444a54640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3440102b30 0x7f34401981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3444a54640 1 -- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34401034f0 msgr2=0x7f344019f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3444a54640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34401034f0 0x7f344019f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3444a54640 1 -- 172.21.15.161:0/1702514314 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344019f890 con 0x7f3440101930 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3444a54640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440197c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3430031610 tx=0x7f34300043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f343dffb640 1 -- 172.21.15.161:0/1702514314 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343003f030 con 0x7f3440101930 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f343dffb640 1 -- 172.21.15.161:0/1702514314 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3430003ce0 con 0x7f3440101930 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f343dffb640 1 -- 172.21.15.161:0/1702514314 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430035070 con 0x7f3440101930 2024-08-19T01:29:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344019fb20 con 0x7f3440101930 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.923+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34401a0050 con 0x7f3440101930 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.924+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3440108d80 con 0x7f3440101930 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.926+0000 7f343dffb640 1 -- 172.21.15.161:0/1702514314 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3430042030 con 0x7f3440101930 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.926+0000 7f343dffb640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34280a2c00 0x7f34280a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.926+0000 7f343dffb640 1 -- 172.21.15.161:0/1702514314 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6723+0+0 (secure 0 0 0) 0x7f34300ef260 con 0x7f3440101930 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.927+0000 7f343ffff640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34280a2c00 0x7f34280a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.927+0000 7f343ffff640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34280a2c00 0x7f34280a50c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f343400bbe0 tx=0x7f343400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:29:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:38.929+0000 7f343dffb640 1 -- 172.21.15.161:0/1702514314 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f34300bc7f0 con 0x7f3440101930 2024-08-19T01:29:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:38 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:29:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:38 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:29:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:38 smithi064 ceph-mon[18515]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-08-19T01:29:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:38 smithi064 ceph-mon[18515]: osd.10 [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] boot 2024-08-19T01:29:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:38 smithi064 ceph-mon[18515]: osdmap e69: 11 total, 11 up, 11 in 2024-08-19T01:29:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:38 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:29:38.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:38 smithi064 ceph-mon[18515]: osdmap e70: 11 total, 11 up, 11 in 2024-08-19T01:29:39.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:39.107+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3440062360 con 0x7f34280a2c00 2024-08-19T01:29:40.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi123 2024-08-19T01:29:40.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi123 to 4272M 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:29:40.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:29:40.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:40.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:40.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:40 smithi161 ceph-mon[21757]: osdmap e71: 11 total, 11 up, 11 in 2024-08-19T01:29:40.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi123 2024-08-19T01:29:40.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi123 to 4272M 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:40 smithi123 ceph-mon[21829]: osdmap e71: 11 total, 11 up, 11 in 2024-08-19T01:29:40.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi123 2024-08-19T01:29:40.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi123 to 4272M 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:40.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:40.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:40.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:40.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:40 smithi064 ceph-mon[18515]: osdmap e71: 11 total, 11 up, 11 in 2024-08-19T01:29:41.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:41 smithi123 ceph-mon[21829]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:41.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:41 smithi123 ceph-mon[21829]: osdmap e72: 11 total, 11 up, 11 in 2024-08-19T01:29:41.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:41 smithi161 ceph-mon[21757]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:41.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:41 smithi161 ceph-mon[21757]: osdmap e72: 11 total, 11 up, 11 in 2024-08-19T01:29:41.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:41 smithi064 ceph-mon[18515]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:41.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:41 smithi064 ceph-mon[18515]: osdmap e72: 11 total, 11 up, 11 in 2024-08-19T01:29:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:42 smithi123 ceph-mon[21829]: osdmap e73: 11 total, 11 up, 11 in 2024-08-19T01:29:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:42 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/3916119170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]: dispatch 2024-08-19T01:29:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:42 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]: dispatch 2024-08-19T01:29:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:42 smithi123 ceph-mon[21829]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]': finished 2024-08-19T01:29:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:42 smithi123 ceph-mon[21829]: osdmap e74: 12 total, 11 up, 12 in 2024-08-19T01:29:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:29:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:42 smithi123 ceph-mon[21829]: from='client.? 172.21.15.161:0/3036641626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:29:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:42 smithi161 ceph-mon[21757]: osdmap e73: 11 total, 11 up, 11 in 2024-08-19T01:29:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:42 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/3916119170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]: dispatch 2024-08-19T01:29:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:42 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]: dispatch 2024-08-19T01:29:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:42 smithi161 ceph-mon[21757]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]': finished 2024-08-19T01:29:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:42 smithi161 ceph-mon[21757]: osdmap e74: 12 total, 11 up, 12 in 2024-08-19T01:29:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:29:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:42 smithi161 ceph-mon[21757]: from='client.? 172.21.15.161:0/3036641626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:29:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:42 smithi064 ceph-mon[18515]: osdmap e73: 11 total, 11 up, 11 in 2024-08-19T01:29:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:42 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/3916119170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]: dispatch 2024-08-19T01:29:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:42 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]: dispatch 2024-08-19T01:29:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:42 smithi064 ceph-mon[18515]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d23aa4f-abcc-4207-aa16-d6e3975b613f"}]': finished 2024-08-19T01:29:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:42 smithi064 ceph-mon[18515]: osdmap e74: 12 total, 11 up, 12 in 2024-08-19T01:29:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:29:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:42 smithi064 ceph-mon[18515]: from='client.? 172.21.15.161:0/3036641626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T01:29:43.919 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:43 smithi123 ceph-mon[21829]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:43 smithi161 ceph-mon[21757]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:43 smithi064 ceph-mon[18515]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 353 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:45.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:45 smithi123 ceph-mon[21829]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-08-19T01:29:45.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:45 smithi161 ceph-mon[21757]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-08-19T01:29:45.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:45 smithi064 ceph-mon[18515]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-08-19T01:29:47.681 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:47 smithi161 ceph-mon[21757]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-19T01:29:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:47 smithi123 ceph-mon[21829]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-19T01:29:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:47 smithi064 ceph-mon[18515]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-19T01:29:49.780 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:49 smithi161 ceph-mon[21757]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-08-19T01:29:49.780 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:49 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-08-19T01:29:49.780 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:49 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:49.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:49 smithi123 ceph-mon[21829]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-08-19T01:29:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:49 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-08-19T01:29:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:49 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:49.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:49 smithi064 ceph-mon[18515]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-08-19T01:29:49.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:49 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-08-19T01:29:49.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:49 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:50.674 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:50 smithi161 ceph-mon[21757]: Deploying daemon osd.11 on smithi161 2024-08-19T01:29:50.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:50 smithi123 ceph-mon[21829]: Deploying daemon osd.11 on smithi161 2024-08-19T01:29:50.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:50 smithi064 ceph-mon[18515]: Deploying daemon osd.11 on smithi161 2024-08-19T01:29:51.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:51 smithi161 ceph-mon[21757]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-08-19T01:29:51.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:51 smithi123 ceph-mon[21829]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-08-19T01:29:51.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:51 smithi064 ceph-mon[18515]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-08-19T01:29:52.799 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:52 smithi161 ceph-mon[21757]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-08-19T01:29:52.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:52 smithi123 ceph-mon[21829]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-08-19T01:29:52.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:52 smithi064 ceph-mon[18515]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-08-19T01:29:54.787 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:54 smithi161 ceph-mon[21757]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:54.787 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:54.787 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:54.787 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:54 smithi123 ceph-mon[21829]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:54.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:54 smithi064 ceph-mon[18515]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-19T01:29:54.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:54.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:54.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:56.757 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:56 smithi161 ceph-mon[21757]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:56.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:56 smithi064 ceph-mon[18515]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:57.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:56 smithi123 ceph-mon[21829]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:57.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.718+0000 7f343dffb640 1 -- 172.21.15.161:0/1702514314 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3440062360 con 0x7f34280a2c00 2024-08-19T01:29:57.721 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 11 on host 'smithi161' 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34280a2c00 msgr2=0x7f34280a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34280a2c00 0x7f34280a50c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f343400bbe0 tx=0x7f343400b390 comp rx=0 tx=0).stop 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 msgr2=0x7f3440197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440197c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3430031610 tx=0x7f34300043b0 comp rx=0 tx=0).stop 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 shutdown_connections 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34280a2c00 0x7f34280a50c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34401034f0 0x7f344019f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3440102b30 0x7f34401981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:57.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 --2- 172.21.15.161:0/1702514314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3440101930 0x7f3440197c70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:57.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 >> 172.21.15.161:0/1702514314 conn(0x7f34400fd0c0 msgr2=0x7f3440105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:29:57.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 shutdown_connections 2024-08-19T01:29:57.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:29:57.721+0000 7f3446cb8640 1 -- 172.21.15.161:0/1702514314 wait complete. 2024-08-19T01:29:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:57.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:57.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:57 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:57 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.360 DEBUG:teuthology.orchestra.run.smithi161:osd.11> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.11.service 2024-08-19T01:29:58.363 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-08-19T01:29:58.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd stat -f json 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:57 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:29:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:29:59.079 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:58 smithi064 ceph-mon[18515]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:59.079 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:58 smithi064 ceph-mon[18515]: from='osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T01:29:59.079 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:58 smithi064 ceph-mon[18515]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T01:29:59.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 -- 172.21.15.64:0/1791980384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee81023c0 msgr2=0x7fcee8108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 --2- 172.21.15.64:0/1791980384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee81023c0 0x7fcee8108c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009be0 tx=0x7fcee002a6c0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 -- 172.21.15.64:0/1791980384 shutdown_connections 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 --2- 172.21.15.64:0/1791980384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee81023c0 0x7fcee8108c50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 --2- 172.21.15.64:0/1791980384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8101a00 0x7fcee8101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 --2- 172.21.15.64:0/1791980384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcee8100800 0x7fcee8100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 -- 172.21.15.64:0/1791980384 >> 172.21.15.64:0/1791980384 conn(0x7fcee80fbfb0 msgr2=0x7fcee80fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 -- 172.21.15.64:0/1791980384 shutdown_connections 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.077+0000 7fcef03b0640 1 -- 172.21.15.64:0/1791980384 wait complete. 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 Processor -- start 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 -- start start 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8100800 0x7fcee8104790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee8101a00 0x7fcee8102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:59.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcee81023c0 0x7fcee8103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcee810c430 con 0x7fcee8101a00 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcee810c2b0 con 0x7fcee8100800 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fcef03b0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcee810c5b0 con 0x7fcee81023c0 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee8101a00 0x7fcee8102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee8101a00 0x7fcee8102de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57498/0 (socket says 172.21.15.64:57498) 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 -- 172.21.15.64:0/231586413 learned_addr learned my addr 172.21.15.64:0/231586413 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceee94d640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcee81023c0 0x7fcee8103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 -- 172.21.15.64:0/231586413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcee81023c0 msgr2=0x7fcee8103320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcee81023c0 0x7fcee8103320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 -- 172.21.15.64:0/231586413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8100800 msgr2=0x7fcee8104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8100800 0x7fcee8104790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.078+0000 7fceed94b640 1 -- 172.21.15.64:0/231586413 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee8103be0 con 0x7fcee8101a00 2024-08-19T01:29:59.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.079+0000 7fceed94b640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee8101a00 0x7fcee8102de0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fced800d950 tx=0x7fced800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.079+0000 7fcedf7fe640 1 -- 172.21.15.64:0/231586413 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8017070 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.079+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee81a1d60 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.079+0000 7fcedf7fe640 1 -- 172.21.15.64:0/231586413 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced8009c30 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.079+0000 7fcedf7fe640 1 -- 172.21.15.64:0/231586413 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced800f6b0 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.079+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee81a2210 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.080+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee81a2440 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.081+0000 7fcedf7fe640 1 -- 172.21.15.64:0/231586413 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fced80109d0 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.082+0000 7fcedf7fe640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fced40a2b80 0x7fced40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.082+0000 7fceee14c640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fced40a2b80 0x7fced40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.082+0000 7fcedf7fe640 1 -- 172.21.15.64:0/231586413 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6831+0+0 (secure 0 0 0) 0x7fced80ce6b0 con 0x7fcee8101a00 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.082+0000 7fceee14c640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fced40a2b80 0x7fced40a5040 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcee8101860 tx=0x7fcee4009660 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:29:59.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.086+0000 7fcedf7fe640 1 -- 172.21.15.64:0/231586413 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fced8009de0 con 0x7fcee8101a00 2024-08-19T01:29:59.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:58 smithi161 ceph-mon[21757]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:59.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:58 smithi161 ceph-mon[21757]: from='osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T01:29:59.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:58 smithi161 ceph-mon[21757]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T01:29:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.263+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fcee81a2730 con 0x7fcee8101a00 2024-08-19T01:29:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.263+0000 7fcedf7fe640 1 -- 172.21.15.64:0/231586413 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) ==== 74+0+133 (secure 0 0 0) 0x7fced809bbe0 con 0x7fcee8101a00 2024-08-19T01:29:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:29:59.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.265+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fced40a2b80 msgr2=0x7fced40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:59.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.265+0000 7fcef03b0640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fced40a2b80 0x7fced40a5040 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcee8101860 tx=0x7fcee4009660 comp rx=0 tx=0).stop 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.265+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee8101a00 msgr2=0x7fcee8102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.265+0000 7fcef03b0640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee8101a00 0x7fcee8102de0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fced800d950 tx=0x7fced800de10 comp rx=0 tx=0).stop 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 shutdown_connections 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fced40a2b80 0x7fced40a5040 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcee81023c0 0x7fcee8103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee8101a00 0x7fcee8102de0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 --2- 172.21.15.64:0/231586413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8100800 0x7fcee8104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 >> 172.21.15.64:0/231586413 conn(0x7fcee80fbfb0 msgr2=0x7fcee8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:29:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 shutdown_connections 2024-08-19T01:29:59.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:29:59.266+0000 7fcef03b0640 1 -- 172.21.15.64:0/231586413 wait complete. 2024-08-19T01:29:59.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:58 smithi123 ceph-mon[21829]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:29:59.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:58 smithi123 ceph-mon[21829]: from='osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T01:29:59.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:58 smithi123 ceph-mon[21829]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T01:30:00.049 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1724030977,"num_in_osds":12,"osd_in_since":1724030981,"num_remapped_pgs":0} 2024-08-19T01:30:00.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:59 smithi064 ceph-mon[18515]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-19T01:30:00.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:59 smithi064 ceph-mon[18515]: osdmap e75: 12 total, 11 up, 12 in 2024-08-19T01:30:00.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:59 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:00.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:59 smithi064 ceph-mon[18515]: from='osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:30:00.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:59 smithi064 ceph-mon[18515]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:30:00.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:29:59 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/231586413' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T01:30:00.296 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-mon[21757]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-19T01:30:00.296 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-mon[21757]: osdmap e75: 12 total, 11 up, 12 in 2024-08-19T01:30:00.296 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:00.296 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-mon[21757]: from='osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:30:00.296 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-mon[21757]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:30:00.296 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/231586413' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T01:30:00.297 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:29:59.961+0000 7f2486bfa640 -1 osd.11 0 waiting for initial osdmap 2024-08-19T01:30:00.297 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:29:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:29:59.973+0000 7f2482a22640 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T01:30:00.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:59 smithi123 ceph-mon[21829]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-19T01:30:00.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:59 smithi123 ceph-mon[21829]: osdmap e75: 12 total, 11 up, 12 in 2024-08-19T01:30:00.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:59 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:00.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:59 smithi123 ceph-mon[21829]: from='osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:30:00.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:59 smithi123 ceph-mon[21829]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-08-19T01:30:00.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:29:59 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/231586413' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T01:30:01.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd stat -f json 2024-08-19T01:30:01.158 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:00 smithi161 ceph-mon[21757]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:30:01.158 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:00 smithi161 ceph-mon[21757]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:30:01.158 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:00 smithi161 ceph-mon[21757]: osdmap e76: 12 total, 11 up, 12 in 2024-08-19T01:30:01.158 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:00 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:01.158 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:00 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:01.158 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:00 smithi161 ceph-mon[21757]: overall HEALTH_OK 2024-08-19T01:30:01.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:00 smithi064 ceph-mon[18515]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:30:01.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:00 smithi064 ceph-mon[18515]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:30:01.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:00 smithi064 ceph-mon[18515]: osdmap e76: 12 total, 11 up, 12 in 2024-08-19T01:30:01.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:00 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:01.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:00 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:01.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:00 smithi064 ceph-mon[18515]: overall HEALTH_OK 2024-08-19T01:30:01.309 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:01.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:00 smithi123 ceph-mon[21829]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:30:01.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:00 smithi123 ceph-mon[21829]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-08-19T01:30:01.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:00 smithi123 ceph-mon[21829]: osdmap e76: 12 total, 11 up, 12 in 2024-08-19T01:30:01.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:00 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:01.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:00 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:01.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:00 smithi123 ceph-mon[21829]: overall HEALTH_OK 2024-08-19T01:30:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.772+0000 7fa75b39a640 1 -- 172.21.15.64:0/1697615832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754104d60 msgr2=0x7fa754107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.772+0000 7fa75b39a640 1 --2- 172.21.15.64:0/1697615832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754104d60 0x7fa754107150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa748009900 tx=0x7fa74802a410 comp rx=0 tx=0).stop 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.772+0000 7fa75b39a640 1 -- 172.21.15.64:0/1697615832 shutdown_connections 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.772+0000 7fa75b39a640 1 --2- 172.21.15.64:0/1697615832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa754107690 0x7fa754109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.772+0000 7fa75b39a640 1 --2- 172.21.15.64:0/1697615832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754104d60 0x7fa754107150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.772+0000 7fa75b39a640 1 --2- 172.21.15.64:0/1697615832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa754102430 0x7fa754104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.772+0000 7fa75b39a640 1 -- 172.21.15.64:0/1697615832 >> 172.21.15.64:0/1697615832 conn(0x7fa7540fbfb0 msgr2=0x7fa7540fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 -- 172.21.15.64:0/1697615832 shutdown_connections 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 -- 172.21.15.64:0/1697615832 wait complete. 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 Processor -- start 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 -- start start 2024-08-19T01:30:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa754102430 0x7fa754101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa754104d60 0x7fa7541003e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754107690 0x7fa754100920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa75410d580 con 0x7fa754107690 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa75410d400 con 0x7fa754104d60 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa75b39a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa75410d700 con 0x7fa754102430 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa759937640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754107690 0x7fa754100920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa759937640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754107690 0x7fa754100920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57510/0 (socket says 172.21.15.64:57510) 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa759937640 1 -- 172.21.15.64:0/885863087 learned_addr learned my addr 172.21.15.64:0/885863087 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa759136640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa754102430 0x7fa754101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.773+0000 7fa758935640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa754104d60 0x7fa7541003e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa759937640 1 -- 172.21.15.64:0/885863087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa754102430 msgr2=0x7fa754101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa759937640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa754102430 0x7fa754101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa759937640 1 -- 172.21.15.64:0/885863087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa754104d60 msgr2=0x7fa7541003e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa759937640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa754104d60 0x7fa7541003e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa759937640 1 -- 172.21.15.64:0/885863087 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7541011e0 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa759937640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754107690 0x7fa754100920 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa74400d750 tx=0x7fa74400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa7527fc640 1 -- 172.21.15.64:0/885863087 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744017070 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7541a1df0 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa7527fc640 1 -- 172.21.15.64:0/885863087 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa744005500 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa7527fc640 1 -- 172.21.15.64:0/885863087 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744004d90 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.774+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7541a2250 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.775+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7540fdaa0 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.776+0000 7fa7527fc640 1 -- 172.21.15.64:0/885863087 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa744004f30 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.777+0000 7fa7527fc640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa73c0a2b80 0x7fa73c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.777+0000 7fa759136640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa73c0a2b80 0x7fa73c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.777+0000 7fa7527fc640 1 -- 172.21.15.64:0/885863087 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7fa7440ce5b0 con 0x7fa754107690 2024-08-19T01:30:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.778+0000 7fa759136640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa73c0a2b80 0x7fa73c0a5040 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa74c005fd0 tx=0x7fa74c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.782+0000 7fa7527fc640 1 -- 172.21.15.64:0/885863087 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa74409b2c0 con 0x7fa754107690 2024-08-19T01:30:01.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.958+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa754062b20 con 0x7fa754107690 2024-08-19T01:30:01.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.958+0000 7fa7527fc640 1 -- 172.21.15.64:0/885863087 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) ==== 74+0+133 (secure 0 0 0) 0x7fa74409f5c0 con 0x7fa754107690 2024-08-19T01:30:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa73c0a2b80 msgr2=0x7fa73c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa73c0a2b80 0x7fa73c0a5040 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa74c005fd0 tx=0x7fa74c009500 comp rx=0 tx=0).stop 2024-08-19T01:30:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754107690 msgr2=0x7fa754100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754107690 0x7fa754100920 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa74400d750 tx=0x7fa74400dc10 comp rx=0 tx=0).stop 2024-08-19T01:30:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 shutdown_connections 2024-08-19T01:30:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa73c0a2b80 0x7fa73c0a5040 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa754107690 0x7fa754100920 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa754104d60 0x7fa7541003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 --2- 172.21.15.64:0/885863087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa754102430 0x7fa754101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.960+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 >> 172.21.15.64:0/885863087 conn(0x7fa7540fbfb0 msgr2=0x7fa754102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.961+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 shutdown_connections 2024-08-19T01:30:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:01.961+0000 7fa75b39a640 1 -- 172.21.15.64:0/885863087 wait complete. 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: purged_snaps scrub starts 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: purged_snaps scrub ok 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] boot 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: osdmap e77: 12 total, 12 up, 12 in 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:02.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:02.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:01 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/885863087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: purged_snaps scrub starts 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: purged_snaps scrub ok 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] boot 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: osdmap e77: 12 total, 12 up, 12 in 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:02.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:02.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:01 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/885863087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T01:30:02.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: purged_snaps scrub starts 2024-08-19T01:30:02.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: purged_snaps scrub ok 2024-08-19T01:30:02.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: osd.11 [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] boot 2024-08-19T01:30:02.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: osdmap e77: 12 total, 12 up, 12 in 2024-08-19T01:30:02.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:02.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:01 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/885863087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T01:30:02.724 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1724031000,"num_in_osds":12,"osd_in_since":1724030981,"num_remapped_pgs":0} 2024-08-19T01:30:02.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:30:02.977 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:02.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:02 smithi064 ceph-mon[18515]: Detected new or changed devices on smithi161 2024-08-19T01:30:02.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:02 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi161 to 4272M 2024-08-19T01:30:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:02 smithi064 ceph-mon[18515]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:30:02.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:02 smithi064 ceph-mon[18515]: osdmap e78: 12 total, 12 up, 12 in 2024-08-19T01:30:03.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:02 smithi123 ceph-mon[21829]: Detected new or changed devices on smithi161 2024-08-19T01:30:03.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:02 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi161 to 4272M 2024-08-19T01:30:03.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:02 smithi123 ceph-mon[21829]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:30:03.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:02 smithi123 ceph-mon[21829]: osdmap e78: 12 total, 12 up, 12 in 2024-08-19T01:30:03.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:02 smithi161 ceph-mon[21757]: Detected new or changed devices on smithi161 2024-08-19T01:30:03.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:02 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi161 to 4272M 2024-08-19T01:30:03.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:02 smithi161 ceph-mon[21757]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-19T01:30:03.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:02 smithi161 ceph-mon[21757]: osdmap e78: 12 total, 12 up, 12 in 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 -- 172.21.15.64:0/3749320602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901007b0 msgr2=0x7f6090100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 --2- 172.21.15.64:0/3749320602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901007b0 0x7f6090100c10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6084009930 tx=0x7f608402a420 comp rx=0 tx=0).stop 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 -- 172.21.15.64:0/3749320602 shutdown_connections 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 --2- 172.21.15.64:0/3749320602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6090101150 0x7f609010d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 --2- 172.21.15.64:0/3749320602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901007b0 0x7f6090100c10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 --2- 172.21.15.64:0/3749320602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60901067b0 0x7f6090106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 -- 172.21.15.64:0/3749320602 >> 172.21.15.64:0/3749320602 conn(0x7f60900fc4d0 msgr2=0x7f60900fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:03.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 -- 172.21.15.64:0/3749320602 shutdown_connections 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 -- 172.21.15.64:0/3749320602 wait complete. 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.429+0000 7f60980be640 1 Processor -- start 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f60980be640 1 -- start start 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f60980be640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60901007b0 0x7f6090198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f60980be640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6090101150 0x7f6090198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f60980be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901067b0 0x7f609019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f60980be640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6090111cd0 con 0x7f60901067b0 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f60980be640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6090111b50 con 0x7f60901007b0 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f60980be640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6090111e50 con 0x7f6090101150 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901067b0 0x7f609019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f6095e5a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60901007b0 0x7f6090198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901067b0 0x7f609019cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57528/0 (socket says 172.21.15.64:57528) 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 -- 172.21.15.64:0/3048185696 learned_addr learned my addr 172.21.15.64:0/3048185696 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:03.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f6095659640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6090101150 0x7f6090198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 -- 172.21.15.64:0/3048185696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6090101150 msgr2=0x7f6090198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6090101150 0x7f6090198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 -- 172.21.15.64:0/3048185696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60901007b0 msgr2=0x7f6090198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60901007b0 0x7f6090198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 -- 172.21.15.64:0/3048185696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609019d1c0 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f6095e5a640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60901007b0 0x7f6090198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.430+0000 7f609665b640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901067b0 0x7f609019cae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f608800cad0 tx=0x7f608800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.431+0000 7f6082ffd640 1 -- 172.21.15.64:0/3048185696 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608800ad60 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.431+0000 7f6082ffd640 1 -- 172.21.15.64:0/3048185696 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6088005020 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.431+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609019d4b0 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.431+0000 7f6082ffd640 1 -- 172.21.15.64:0/3048185696 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608800f6b0 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.431+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60901a4d90 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.432+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6090100c10 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.433+0000 7f6082ffd640 1 -- 172.21.15.64:0/3048185696 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6088005560 con 0x7f60901067b0 2024-08-19T01:30:03.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.433+0000 7f6082ffd640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f607c0a2b80 0x7f607c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:03.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.433+0000 7f6095e5a640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f607c0a2b80 0x7f607c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:03.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.433+0000 7f6082ffd640 1 -- 172.21.15.64:0/3048185696 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f60880cec40 con 0x7f60901067b0 2024-08-19T01:30:03.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.434+0000 7f6095e5a640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f607c0a2b80 0x7f607c0a5040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f608c009d60 tx=0x7f608c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:03.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.437+0000 7f6082ffd640 1 -- 172.21.15.64:0/3048185696 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6088014030 con 0x7f60901067b0 2024-08-19T01:30:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.613+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60901087b0 con 0x7f60901067b0 2024-08-19T01:30:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.614+0000 7f6082ffd640 1 -- 172.21.15.64:0/3048185696 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+18445 (secure 0 0 0) 0x7f608809c040 con 0x7f60901067b0 2024-08-19T01:30:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:03.617 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":78,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:30:01.963919+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:30:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f607c0a2b80 msgr2=0x7f607c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f607c0a2b80 0x7f607c0a5040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f608c009d60 tx=0x7f608c009110 comp rx=0 tx=0).stop 2024-08-19T01:30:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901067b0 msgr2=0x7f609019cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901067b0 0x7f609019cae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f608800cad0 tx=0x7f608800ced0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 shutdown_connections 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f607c0a2b80 0x7f607c0a5040 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60901067b0 0x7f609019cae0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6090101150 0x7f6090198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 --2- 172.21.15.64:0/3048185696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60901007b0 0x7f6090198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 >> 172.21.15.64:0/3048185696 conn(0x7f60900fc4d0 msgr2=0x7f60900fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 shutdown_connections 2024-08-19T01:30:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:03.617+0000 7f60980be640 1 -- 172.21.15.64:0/3048185696 wait complete. 2024-08-19T01:30:04.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:03 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3048185696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:30:04.343 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T01:26:37.913262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-19T01:30:04.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-08-19T01:30:04.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:03 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3048185696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:30:04.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:03 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3048185696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:30:04.595 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:05.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/147050068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd410bea0 msgr2=0x7fdcd410e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:05.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/147050068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd410bea0 0x7fdcd410e340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009ca0 tx=0x7fdccc02a950 comp rx=0 tx=0).stop 2024-08-19T01:30:05.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/147050068 shutdown_connections 2024-08-19T01:30:05.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/147050068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd410bea0 0x7fdcd410e340 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/147050068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcd4103390 0x7fdcd410b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/147050068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcd4102a70 0x7fdcd4102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/147050068 >> 172.21.15.64:0/147050068 conn(0x7fdcd40fc4d0 msgr2=0x7fdcd40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/147050068 shutdown_connections 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.022+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/147050068 wait complete. 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 Processor -- start 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 -- start start 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd4102a70 0x7fdcd419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcd4103390 0x7fdcd419b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcd410bea0 0x7fdcd410a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4111ce0 con 0x7fdcd4102a70 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4111b60 con 0x7fdcd410bea0 2024-08-19T01:30:05.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcdc1a5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4111e60 con 0x7fdcd4103390 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcd9f41640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd4102a70 0x7fdcd419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcd9f41640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd4102a70 0x7fdcd419abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57544/0 (socket says 172.21.15.64:57544) 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.023+0000 7fdcd9f41640 1 -- 172.21.15.64:0/974717759 learned_addr learned my addr 172.21.15.64:0/974717759 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcda742640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcd410bea0 0x7fdcd410a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcd9740640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcd4103390 0x7fdcd419b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcd9f41640 1 -- 172.21.15.64:0/974717759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcd4103390 msgr2=0x7fdcd419b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcd9f41640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcd4103390 0x7fdcd419b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcd9f41640 1 -- 172.21.15.64:0/974717759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcd410bea0 msgr2=0x7fdcd410a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcd9f41640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcd410bea0 0x7fdcd410a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcd9f41640 1 -- 172.21.15.64:0/974717759 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd410b1e0 con 0x7fdcd4102a70 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcd9f41640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd4102a70 0x7fdcd419abf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c920 tx=0x7fdcd000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdccaffd640 1 -- 172.21.15.64:0/974717759 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0007c70 con 0x7fdcd4102a70 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd410b4d0 con 0x7fdcd4102a70 2024-08-19T01:30:05.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdccaffd640 1 -- 172.21.15.64:0/974717759 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd0007e10 con 0x7fdcd4102a70 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdccaffd640 1 -- 172.21.15.64:0/974717759 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0018890 con 0x7fdcd4102a70 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.024+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd419ead0 con 0x7fdcd4102a70 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.025+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd406ada0 con 0x7fdcd4102a70 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.026+0000 7fdccaffd640 1 -- 172.21.15.64:0/974717759 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdcd0018a70 con 0x7fdcd4102a70 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.027+0000 7fdccaffd640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc00a2b80 0x7fdcc00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.027+0000 7fdcd9740640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc00a2b80 0x7fdcc00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.027+0000 7fdccaffd640 1 -- 172.21.15.64:0/974717759 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fdcd00cec40 con 0x7fdcd4102a70 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.027+0000 7fdcd9740640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc00a2b80 0x7fdcc00a5040 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdcd41974d0 tx=0x7fdcc4009060 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:05.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.031+0000 7fdccaffd640 1 -- 172.21.15.64:0/974717759 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fdcd0014030 con 0x7fdcd4102a70 2024-08-19T01:30:05.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.204+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdcd419ed90 con 0x7fdcd4102a70 2024-08-19T01:30:05.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.204+0000 7fdccaffd640 1 -- 172.21.15.64:0/974717759 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7fdcd009c040 con 0x7fdcd4102a70 2024-08-19T01:30:05.206 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-08-19T01:30:05.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc00a2b80 msgr2=0x7fdcc00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc00a2b80 0x7fdcc00a5040 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdcd41974d0 tx=0x7fdcc4009060 comp rx=0 tx=0).stop 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd4102a70 msgr2=0x7fdcd419abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd4102a70 0x7fdcd419abf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c920 tx=0x7fdcd000cde0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 shutdown_connections 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdcc00a2b80 0x7fdcc00a5040 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcd410bea0 0x7fdcd410a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcd4103390 0x7fdcd419b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 --2- 172.21.15.64:0/974717759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcd4102a70 0x7fdcd419abf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 >> 172.21.15.64:0/974717759 conn(0x7fdcd40fc4d0 msgr2=0x7fdcd410d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 shutdown_connections 2024-08-19T01:30:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:05.206+0000 7fdcdc1a5640 1 -- 172.21.15.64:0/974717759 wait complete. 2024-08-19T01:30:05.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:04 smithi064 ceph-mon[18515]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:04 smithi123 ceph-mon[21829]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:05.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:04 smithi161 ceph-mon[21757]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:05.891 INFO:tasks.cephadm:Adding mds.a on smithi064 2024-08-19T01:30:05.891 INFO:tasks.cephadm:Adding mds.d on smithi064 2024-08-19T01:30:05.891 INFO:tasks.cephadm:Adding mds.g on smithi064 2024-08-19T01:30:05.891 INFO:tasks.cephadm:Adding mds.j on smithi064 2024-08-19T01:30:05.891 INFO:tasks.cephadm:Adding mds.b on smithi123 2024-08-19T01:30:05.891 INFO:tasks.cephadm:Adding mds.e on smithi123 2024-08-19T01:30:05.892 INFO:tasks.cephadm:Adding mds.h on smithi123 2024-08-19T01:30:05.892 INFO:tasks.cephadm:Adding mds.k on smithi123 2024-08-19T01:30:05.892 INFO:tasks.cephadm:Adding mds.c on smithi161 2024-08-19T01:30:05.892 INFO:tasks.cephadm:Adding mds.f on smithi161 2024-08-19T01:30:05.892 INFO:tasks.cephadm:Adding mds.i on smithi161 2024-08-19T01:30:05.892 INFO:tasks.cephadm:Adding mds.l on smithi161 2024-08-19T01:30:05.892 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph orch apply mds all '12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l' 2024-08-19T01:30:06.158 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:30:06.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:05 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/974717759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T01:30:06.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:05 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/974717759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T01:30:06.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:05 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/974717759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T01:30:06.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 -- 172.21.15.161:0/2400184234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70102b30 msgr2=0x7f3d70102fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:06.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2400184234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70102b30 0x7f3d70102fb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009900 tx=0x7f3d6402a410 comp rx=0 tx=0).stop 2024-08-19T01:30:06.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 -- 172.21.15.161:0/2400184234 shutdown_connections 2024-08-19T01:30:06.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2400184234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d701034f0 0x7f3d70109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2400184234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70102b30 0x7f3d70102fb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2400184234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d70101930 0x7f3d70101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 -- 172.21.15.161:0/2400184234 >> 172.21.15.161:0/2400184234 conn(0x7f3d700fd0c0 msgr2=0x7f3d700ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 -- 172.21.15.161:0/2400184234 shutdown_connections 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 -- 172.21.15.161:0/2400184234 wait complete. 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.703+0000 7f3d760dd640 1 Processor -- start 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d760dd640 1 -- start start 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d760dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70101930 0x7f3d70197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d760dd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d70102b30 0x7f3d70198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d760dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d701034f0 0x7f3d7019f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d760dd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7010d590 con 0x7f3d70102b30 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d760dd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7010d410 con 0x7f3d701034f0 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d760dd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7010d710 con 0x7f3d70101930 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70101930 0x7f3d70197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70101930 0x7f3d70197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38322/0 (socket says 172.21.15.161:38322) 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 -- 172.21.15.161:0/2512065752 learned_addr learned my addr 172.21.15.161:0/2512065752 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6effd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d70102b30 0x7f3d70198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6ffff640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d701034f0 0x7f3d7019f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:06.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 -- 172.21.15.161:0/2512065752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d701034f0 msgr2=0x7f3d7019f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:06.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d701034f0 0x7f3d7019f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 -- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d70102b30 msgr2=0x7f3d70198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:06.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d70102b30 0x7f3d70198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6f7fe640 1 -- 172.21.15.161:0/2512065752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7019f850 con 0x7f3d70101930 2024-08-19T01:30:06.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.704+0000 7f3d6effd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d70102b30 0x7f3d70198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:30:06.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.705+0000 7f3d6f7fe640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70101930 0x7f3d70197c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d950 tx=0x7f3d6000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:06.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.705+0000 7f3d6cff9640 1 -- 172.21.15.161:0/2512065752 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60017020 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.705+0000 7f3d6cff9640 1 -- 172.21.15.161:0/2512065752 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d60009c30 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.705+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7019fb40 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.705+0000 7f3d6cff9640 1 -- 172.21.15.161:0/2512065752 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000f720 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.705+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d701a0080 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.706+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d70108d80 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.707+0000 7f3d6cff9640 1 -- 172.21.15.161:0/2512065752 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3d6001a020 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.708+0000 7f3d6cff9640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3d580a2b80 0x7f3d580a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.708+0000 7f3d6cff9640 1 -- 172.21.15.161:0/2512065752 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f3d600cebc0 con 0x7f3d70101930 2024-08-19T01:30:06.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.708+0000 7f3d6effd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3d580a2b80 0x7f3d580a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:06.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.708+0000 7f3d6effd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3d580a2b80 0x7f3d580a5040 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d640095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:06.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.712+0000 7f3d6cff9640 1 -- 172.21.15.161:0/2512065752 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f3d60021020 con 0x7f3d70101930 2024-08-19T01:30:06.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.886+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}) -- 0x7f3d70062460 con 0x7f3d580a2b80 2024-08-19T01:30:06.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.894+0000 7f3d6cff9640 1 -- 172.21.15.161:0/2512065752 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f3d70062460 con 0x7f3d580a2b80 2024-08-19T01:30:06.897 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mds.all update... 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3d580a2b80 msgr2=0x7f3d580a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3d580a2b80 0x7f3d580a5040 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d640095f0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70101930 msgr2=0x7f3d70197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70101930 0x7f3d70197c30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d950 tx=0x7f3d6000de10 comp rx=0 tx=0).stop 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 shutdown_connections 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3d580a2b80 0x7f3d580a5040 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d701034f0 0x7f3d7019f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d70102b30 0x7f3d70198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 --2- 172.21.15.161:0/2512065752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d70101930 0x7f3d70197c30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 >> 172.21.15.161:0/2512065752 conn(0x7f3d700fd0c0 msgr2=0x7f3d70105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 shutdown_connections 2024-08-19T01:30:06.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-08-19T01:30:06.896+0000 7f3d760dd640 1 -- 172.21.15.161:0/2512065752 wait complete. 2024-08-19T01:30:07.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:07.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:07.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:07.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:07.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:07.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:07.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:07.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:07.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:06 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:07.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:06 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:07.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:07.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:07.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:07.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:07.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:07.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:07.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:07.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:07.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:06 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:07.712 DEBUG:teuthology.orchestra.run.smithi064:mds.a> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.a.service 2024-08-19T01:30:07.715 DEBUG:teuthology.orchestra.run.smithi064:mds.d> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.d.service 2024-08-19T01:30:07.757 DEBUG:teuthology.orchestra.run.smithi064:mds.g> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.g.service 2024-08-19T01:30:07.800 DEBUG:teuthology.orchestra.run.smithi064:mds.j> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.j.service 2024-08-19T01:30:07.844 DEBUG:teuthology.orchestra.run.smithi123:mds.b> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.b.service 2024-08-19T01:30:07.847 DEBUG:teuthology.orchestra.run.smithi123:mds.e> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.e.service 2024-08-19T01:30:07.890 DEBUG:teuthology.orchestra.run.smithi123:mds.h> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.h.service 2024-08-19T01:30:07.936 DEBUG:teuthology.orchestra.run.smithi123:mds.k> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.k.service 2024-08-19T01:30:07.979 DEBUG:teuthology.orchestra.run.smithi161:mds.c> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.c.service 2024-08-19T01:30:07.982 DEBUG:teuthology.orchestra.run.smithi161:mds.f> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.f.service 2024-08-19T01:30:07.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:07 smithi064 ceph-mon[18515]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:30:07.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:07 smithi064 ceph-mon[18515]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-08-19T01:30:07.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:07 smithi064 ceph-mon[18515]: Deploying daemon mds.l on smithi161 2024-08-19T01:30:08.025 DEBUG:teuthology.orchestra.run.smithi161:mds.i> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.i.service 2024-08-19T01:30:08.068 DEBUG:teuthology.orchestra.run.smithi161:mds.l> sudo journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.l.service 2024-08-19T01:30:08.111 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-08-19T01:30:08.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:08.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:07 smithi161 ceph-mon[21757]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:30:08.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:07 smithi161 ceph-mon[21757]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-08-19T01:30:08.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:07 smithi161 ceph-mon[21757]: Deploying daemon mds.l on smithi161 2024-08-19T01:30:08.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:07 smithi123 ceph-mon[21829]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:30:08.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:07 smithi123 ceph-mon[21829]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-08-19T01:30:08.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:07 smithi123 ceph-mon[21829]: Deploying daemon mds.l on smithi161 2024-08-19T01:30:08.372 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.847+0000 7f80d1656640 1 -- 172.21.15.64:0/2685279274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 msgr2=0x7f80cc107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.847+0000 7f80d1656640 1 --2- 172.21.15.64:0/2685279274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc107c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009cd0 tx=0x7f80bc02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:30:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 -- 172.21.15.64:0/2685279274 shutdown_connections 2024-08-19T01:30:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 --2- 172.21.15.64:0/2685279274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80cc102250 0x7f80cc10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 --2- 172.21.15.64:0/2685279274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80cc1018b0 0x7f80cc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 --2- 172.21.15.64:0/2685279274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc107c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 -- 172.21.15.64:0/2685279274 >> 172.21.15.64:0/2685279274 conn(0x7f80cc0fd5f0 msgr2=0x7f80cc0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 -- 172.21.15.64:0/2685279274 shutdown_connections 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 -- 172.21.15.64:0/2685279274 wait complete. 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 Processor -- start 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.848+0000 7f80d1656640 1 -- start start 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80d1656640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80cc1018b0 0x7f80cc198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80d1656640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80cc102250 0x7f80cc198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80d1656640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80d1656640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc111d50 con 0x7f80cc1078b0 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80d1656640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc111bd0 con 0x7f80cc1018b0 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80d1656640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc111ed0 con 0x7f80cc102250 2024-08-19T01:30:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc19cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51738/0 (socket says 172.21.15.64:51738) 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 -- 172.21.15.64:0/3752940168 learned_addr learned my addr 172.21.15.64:0/3752940168 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80caffd640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80cc1018b0 0x7f80cc198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80ca7fc640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80cc102250 0x7f80cc198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 -- 172.21.15.64:0/3752940168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80cc102250 msgr2=0x7f80cc198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80cc102250 0x7f80cc198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 -- 172.21.15.64:0/3752940168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80cc1018b0 msgr2=0x7f80cc198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80cc1018b0 0x7f80cc198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 -- 172.21.15.64:0/3752940168 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80cc19d300 con 0x7f80cc1078b0 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80ca7fc640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80cc102250 0x7f80cc198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80cb7fe640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc19cc20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009fd0 tx=0x7f80c400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.849+0000 7f80b3fff640 1 -- 172.21.15.64:0/3752940168 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4011830 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.850+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80cc19d5f0 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.850+0000 7f80b3fff640 1 -- 172.21.15.64:0/3752940168 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c4016070 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.850+0000 7f80b3fff640 1 -- 172.21.15.64:0/3752940168 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c400f2e0 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.850+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80cc1a4ed0 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.851+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80cc101d90 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.852+0000 7f80b3fff640 1 -- 172.21.15.64:0/3752940168 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f80c40119d0 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.853+0000 7f80b3fff640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f80b40a2b80 0x7f80b40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.853+0000 7f80caffd640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f80b40a2b80 0x7f80b40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.853+0000 7f80b3fff640 1 -- 172.21.15.64:0/3752940168 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f80c40ce230 con 0x7f80cc1078b0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.853+0000 7f80caffd640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f80b40a2b80 0x7f80b40a5040 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f80bc006010 tx=0x7f80bc009900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:08.857+0000 7f80b3fff640 1 -- 172.21.15.64:0/3752940168 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f80c409b630 con 0x7f80cc1078b0 2024-08-19T01:30:09.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.067+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80cc109e80 con 0x7f80cc1078b0 2024-08-19T01:30:09.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:08 smithi064 ceph-mon[18515]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:09.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.067+0000 7f80b3fff640 1 -- 172.21.15.64:0/3752940168 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7f80c409af30 con 0x7f80cc1078b0 2024-08-19T01:30:09.069 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:09.069 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":1,"btime":"2024-08-19T01:23:48:740524+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-08-19T01:30:09.069 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 1 2024-08-19T01:30:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f80b40a2b80 msgr2=0x7f80b40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f80b40a2b80 0x7f80b40a5040 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f80bc006010 tx=0x7f80bc009900 comp rx=0 tx=0).stop 2024-08-19T01:30:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 msgr2=0x7f80cc19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc19cc20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009fd0 tx=0x7f80c400b8e0 comp rx=0 tx=0).stop 2024-08-19T01:30:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 shutdown_connections 2024-08-19T01:30:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f80b40a2b80 0x7f80b40a5040 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80cc1078b0 0x7f80cc19cc20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80cc102250 0x7f80cc198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 --2- 172.21.15.64:0/3752940168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80cc1018b0 0x7f80cc198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.069+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 >> 172.21.15.64:0/3752940168 conn(0x7f80cc0fd5f0 msgr2=0x7f80cc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.070+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 shutdown_connections 2024-08-19T01:30:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:09.070+0000 7f80d1656640 1 -- 172.21.15.64:0/3752940168 wait complete. 2024-08-19T01:30:09.157 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:08 smithi161 ceph-mon[21757]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:08 smithi123 ceph-mon[21829]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:09.449 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 systemd[1]: Starting Ceph mds.l for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:09.816 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 podman[51313]: 2024-08-19 01:30:09.461734344 +0000 UTC m=+0.017049770 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:09.816 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 podman[51313]: 2024-08-19 01:30:09.587315587 +0000 UTC m=+0.142630999 container create 3427f5f088bb8d98c5d5e7697ffeff70b84efdbebc750c8a27d58ea64a04234a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-de09dd0) 2024-08-19T01:30:09.816 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 podman[51313]: 2024-08-19 01:30:09.814942035 +0000 UTC m=+0.370257460 container init 3427f5f088bb8d98c5d5e7697ffeff70b84efdbebc750c8a27d58ea64a04234a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-08-19T01:30:10.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3752940168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:10.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6832/4151238815,v1:172.21.15.161:6833/4151238815] up:boot 2024-08-19T01:30:10.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: fsmap 1 up:standby 2024-08-19T01:30:10.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-08-19T01:30:10.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 podman[51313]: 2024-08-19 01:30:09.817972501 +0000 UTC m=+0.373287914 container start 3427f5f088bb8d98c5d5e7697ffeff70b84efdbebc750c8a27d58ea64a04234a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-mds[51332]: -- 172.21.15.161:0/1448432398 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df8dccd860 con 0x55df8dcab000 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l[51328]: starting mds.l at 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 bash[51313]: 3427f5f088bb8d98c5d5e7697ffeff70b84efdbebc750c8a27d58ea64a04234a 2024-08-19T01:30:10.200 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:30:09 smithi161 systemd[1]: Started Ceph mds.l for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:10.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3752940168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6832/4151238815,v1:172.21.15.161:6833/4151238815] up:boot 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: fsmap 1 up:standby 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:10.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:09 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3752940168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6832/4151238815,v1:172.21.15.161:6833/4151238815] up:boot 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: fsmap 1 up:standby 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:10.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:09 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:11.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:10 smithi123 ceph-mon[21829]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:11.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:10 smithi123 ceph-mon[21829]: Deploying daemon mds.k on smithi123 2024-08-19T01:30:11.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:10 smithi161 ceph-mon[21757]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:11.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:10 smithi161 ceph-mon[21757]: Deploying daemon mds.k on smithi123 2024-08-19T01:30:11.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:10 smithi064 ceph-mon[18515]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:11.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:10 smithi064 ceph-mon[18515]: Deploying daemon mds.k on smithi123 2024-08-19T01:30:11.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:11.977 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:12.166 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 systemd[1]: Starting Ceph mds.k for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 -- 172.21.15.64:0/287547313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a81018b0 msgr2=0x7f75a8101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 --2- 172.21.15.64:0/287547313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a81018b0 0x7f75a8101d10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f759c009930 tx=0x7f759c02a420 comp rx=0 tx=0).stop 2024-08-19T01:30:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 -- 172.21.15.64:0/287547313 shutdown_connections 2024-08-19T01:30:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 --2- 172.21.15.64:0/287547313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75a8102250 0x7f75a810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 --2- 172.21.15.64:0/287547313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a81018b0 0x7f75a8101d10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 --2- 172.21.15.64:0/287547313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75a81078b0 0x7f75a8107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 -- 172.21.15.64:0/287547313 >> 172.21.15.64:0/287547313 conn(0x7f75a80fd5f0 msgr2=0x7f75a80ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 -- 172.21.15.64:0/287547313 shutdown_connections 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.463+0000 7f75ae43f640 1 -- 172.21.15.64:0/287547313 wait complete. 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 Processor -- start 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 -- start start 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75a81018b0 0x7f75a8198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a8102250 0x7f75a8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75a81078b0 0x7f75a819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8111cf0 con 0x7f75a8102250 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8111b70 con 0x7f75a81018b0 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ae43f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8111e70 con 0x7f75a81078b0 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a8102250 0x7f75a8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a8102250 0x7f75a8198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51754/0 (socket says 172.21.15.64:51754) 2024-08-19T01:30:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 -- 172.21.15.64:0/3236701442 learned_addr learned my addr 172.21.15.64:0/3236701442 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a7fff640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75a81018b0 0x7f75a8198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75ac9dc640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75a81078b0 0x7f75a819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 -- 172.21.15.64:0/3236701442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75a81078b0 msgr2=0x7f75a819cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75a81078b0 0x7f75a819cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 -- 172.21.15.64:0/3236701442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75a81018b0 msgr2=0x7f75a8198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75a81018b0 0x7f75a8198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.464+0000 7f75a77fe640 1 -- 172.21.15.64:0/3236701442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a819d1e0 con 0x7f75a8102250 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.465+0000 7f75a77fe640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a8102250 0x7f75a8198770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f759c009230 tx=0x7f759c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.465+0000 7f75a57fa640 1 -- 172.21.15.64:0/3236701442 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c03f030 con 0x7f75a8102250 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.465+0000 7f75a57fa640 1 -- 172.21.15.64:0/3236701442 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f759c003ad0 con 0x7f75a8102250 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.465+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a819d470 con 0x7f75a8102250 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.465+0000 7f75a57fa640 1 -- 172.21.15.64:0/3236701442 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c035070 con 0x7f75a8102250 2024-08-19T01:30:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.465+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a81a4db0 con 0x7f75a8102250 2024-08-19T01:30:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.466+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a8102ff0 con 0x7f75a8102250 2024-08-19T01:30:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.467+0000 7f75a57fa640 1 -- 172.21.15.64:0/3236701442 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f759c042030 con 0x7f75a8102250 2024-08-19T01:30:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.468+0000 7f75a57fa640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f75900a2b80 0x7f75900a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.468+0000 7f75a7fff640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f75900a2b80 0x7f75900a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.468+0000 7f75a57fa640 1 -- 172.21.15.64:0/3236701442 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f759c0ef420 con 0x7f75a8102250 2024-08-19T01:30:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.468+0000 7f75a7fff640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f75900a2b80 0x7f75900a5040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7598005fd0 tx=0x7f759800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.471+0000 7f75a57fa640 1 -- 172.21.15.64:0/3236701442 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f759c0bc1d0 con 0x7f75a8102250 2024-08-19T01:30:12.528 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 podman[50792]: 2024-08-19 01:30:12.151246014 +0000 UTC m=+0.016928429 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:12.528 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 podman[50792]: 2024-08-19 01:30:12.283211098 +0000 UTC m=+0.148893498 container create 0bd66baa8cf233531bc3d9484eba788f7844223fc134167d39bc97fe8931bccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-19T01:30:12.528 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 podman[50792]: 2024-08-19 01:30:12.52766433 +0000 UTC m=+0.393346734 container init 0bd66baa8cf233531bc3d9484eba788f7844223fc134167d39bc97fe8931bccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0) 2024-08-19T01:30:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.682+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75a8101d10 con 0x7f75a8102250 2024-08-19T01:30:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.683+0000 7f75a57fa640 1 -- 172.21.15.64:0/3236701442 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7f75a8101d10 con 0x7f75a8102250 2024-08-19T01:30:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:12.685 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":2,"btime":"2024-08-19T01:30:09:846404+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-08-19T01:30:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 2 2024-08-19T01:30:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f75900a2b80 msgr2=0x7f75900a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f75900a2b80 0x7f75900a5040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7598005fd0 tx=0x7f759800b390 comp rx=0 tx=0).stop 2024-08-19T01:30:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a8102250 msgr2=0x7f75a8198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a8102250 0x7f75a8198770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f759c009230 tx=0x7f759c004270 comp rx=0 tx=0).stop 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 shutdown_connections 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f75900a2b80 0x7f75900a5040 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75a81078b0 0x7f75a819cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75a8102250 0x7f75a8198770 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 --2- 172.21.15.64:0/3236701442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75a81018b0 0x7f75a8198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 >> 172.21.15.64:0/3236701442 conn(0x7f75a80fd5f0 msgr2=0x7f75a80fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 shutdown_connections 2024-08-19T01:30:12.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:12.686+0000 7f75ae43f640 1 -- 172.21.15.64:0/3236701442 wait complete. 2024-08-19T01:30:12.916 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 podman[50792]: 2024-08-19 01:30:12.53056028 +0000 UTC m=+0.396242688 container start 0bd66baa8cf233531bc3d9484eba788f7844223fc134167d39bc97fe8931bccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-08-19T01:30:12.916 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 ceph-mds[50812]: -- 172.21.15.123:0/1137447810 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564571365c20 con 0x564571343400 2024-08-19T01:30:12.916 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k[50808]: starting mds.k at 2024-08-19T01:30:12.916 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 bash[50792]: 0bd66baa8cf233531bc3d9484eba788f7844223fc134167d39bc97fe8931bccc 2024-08-19T01:30:12.916 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:30:12 smithi123 systemd[1]: Started Ceph mds.k for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3236701442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:13.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:13 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3236701442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:13.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:13 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:13.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:13.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:13.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3236701442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:13.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:13.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:13.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:13 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:14.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:14 smithi064 ceph-mon[18515]: Deploying daemon mds.j on smithi064 2024-08-19T01:30:14.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:14 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6832/856914093,v1:172.21.15.123:6833/856914093] up:boot 2024-08-19T01:30:14.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:14 smithi064 ceph-mon[18515]: fsmap 2 up:standby 2024-08-19T01:30:14.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:14 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-08-19T01:30:14.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:14 smithi123 ceph-mon[21829]: Deploying daemon mds.j on smithi064 2024-08-19T01:30:14.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:14 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6832/856914093,v1:172.21.15.123:6833/856914093] up:boot 2024-08-19T01:30:14.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:14 smithi123 ceph-mon[21829]: fsmap 2 up:standby 2024-08-19T01:30:14.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:14 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-08-19T01:30:14.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:14 smithi161 ceph-mon[21757]: Deploying daemon mds.j on smithi064 2024-08-19T01:30:14.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:14 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6832/856914093,v1:172.21.15.123:6833/856914093] up:boot 2024-08-19T01:30:14.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:14 smithi161 ceph-mon[21757]: fsmap 2 up:standby 2024-08-19T01:30:14.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:14 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-08-19T01:30:15.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:15 smithi064 ceph-mon[18515]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:15.235 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 systemd[1]: Starting Ceph mds.j for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:15.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:15.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:15 smithi123 ceph-mon[21829]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:15.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:15 smithi161 ceph-mon[21757]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:15.620 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 podman[54958]: 2024-08-19 01:30:15.283363113 +0000 UTC m=+0.017397115 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:15.621 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 podman[54958]: 2024-08-19 01:30:15.391708056 +0000 UTC m=+0.125742040 container create 0b480a3d658ec620763bb4289c4124531ba4105994c71a9726eb524eeef4bf2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-08-19T01:30:15.621 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 podman[54958]: 2024-08-19 01:30:15.619543451 +0000 UTC m=+0.353577436 container init 0b480a3d658ec620763bb4289c4124531ba4105994c71a9726eb524eeef4bf2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-19T01:30:15.768 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:15.892 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 podman[54958]: 2024-08-19 01:30:15.622757041 +0000 UTC m=+0.356791026 container start 0b480a3d658ec620763bb4289c4124531ba4105994c71a9726eb524eeef4bf2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-19T01:30:15.892 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 ceph-mds[55019]: -- 172.21.15.64:0/3788795032 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558303573860 con 0x558303551400 2024-08-19T01:30:15.892 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j[54997]: starting mds.j at 2024-08-19T01:30:15.892 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 bash[54958]: 0b480a3d658ec620763bb4289c4124531ba4105994c71a9726eb524eeef4bf2a 2024-08-19T01:30:15.892 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:30:15 smithi064 systemd[1]: Started Ceph mds.j for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:16.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 -- 172.21.15.64:0/3824178821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 msgr2=0x7f5f4010e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:16.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/3824178821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4010e340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009ca0 tx=0x7f5f3802a950 comp rx=0 tx=0).stop 2024-08-19T01:30:16.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 -- 172.21.15.64:0/3824178821 shutdown_connections 2024-08-19T01:30:16.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/3824178821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4010e340 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/3824178821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f40103430 0x7f5f4010b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/3824178821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f40102b10 0x7f5f40102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 -- 172.21.15.64:0/3824178821 >> 172.21.15.64:0/3824178821 conn(0x7f5f400fc4d0 msgr2=0x7f5f400fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 -- 172.21.15.64:0/3824178821 shutdown_connections 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.246+0000 7f5f46ee8640 1 -- 172.21.15.64:0/3824178821 wait complete. 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 Processor -- start 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 -- start start 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f40102b10 0x7f5f4019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f40103430 0x7f5f4019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f40111ce0 con 0x7f5f4010beb0 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f40111b60 con 0x7f5f40103430 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f46ee8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5f40111e60 con 0x7f5f40102b10 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f45485640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f45485640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4019ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51780/0 (socket says 172.21.15.64:51780) 2024-08-19T01:30:16.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f45485640 1 -- 172.21.15.64:0/2374823648 learned_addr learned my addr 172.21.15.64:0/2374823648 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f37fff640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f40103430 0x7f5f4019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f44c84640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f40102b10 0x7f5f4019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f45485640 1 -- 172.21.15.64:0/2374823648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f40102b10 msgr2=0x7f5f4019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f45485640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f40102b10 0x7f5f4019a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.247+0000 7f5f45485640 1 -- 172.21.15.64:0/2374823648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f40103430 msgr2=0x7f5f4019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f45485640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f40103430 0x7f5f4019a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f45485640 1 -- 172.21.15.64:0/2374823648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4019f330 con 0x7f5f4010beb0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f37fff640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f40103430 0x7f5f4019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f45485640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4019ec50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5f3802ae70 tx=0x7f5f3802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f35ffb640 1 -- 172.21.15.64:0/2374823648 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f38034040 con 0x7f5f4010beb0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f35ffb640 1 -- 172.21.15.64:0/2374823648 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f38030920 con 0x7f5f4010beb0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4019f5c0 con 0x7f5f4010beb0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f35ffb640 1 -- 172.21.15.64:0/2374823648 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f380047b0 con 0x7f5f4010beb0 2024-08-19T01:30:16.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.248+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f401a6f00 con 0x7f5f4010beb0 2024-08-19T01:30:16.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.249+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f40102ef0 con 0x7f5f4010beb0 2024-08-19T01:30:16.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.250+0000 7f5f35ffb640 1 -- 172.21.15.64:0/2374823648 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5f38037020 con 0x7f5f4010beb0 2024-08-19T01:30:16.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.251+0000 7f5f35ffb640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5f280a2b80 0x7f5f280a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:16.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.251+0000 7f5f44c84640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5f280a2b80 0x7f5f280a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:16.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.251+0000 7f5f35ffb640 1 -- 172.21.15.64:0/2374823648 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f5f380efa20 con 0x7f5f4010beb0 2024-08-19T01:30:16.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.251+0000 7f5f44c84640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5f280a2b80 0x7f5f280a5040 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c005fd0 tx=0x7f5f3c00b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:16.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.255+0000 7f5f35ffb640 1 -- 172.21.15.64:0/2374823648 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f5f380bce20 con 0x7f5f4010beb0 2024-08-19T01:30:16.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.473+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f401a70d0 con 0x7f5f4010beb0 2024-08-19T01:30:16.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.473+0000 7f5f35ffb640 1 -- 172.21.15.64:0/2374823648 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3088 (secure 0 0 0) 0x7f5f380bc720 con 0x7f5f4010beb0 2024-08-19T01:30:16.475 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:16.475 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":4,"btime":"2024-08-19T01:30:16:009943+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-08-19T01:30:16.475 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 4 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5f280a2b80 msgr2=0x7f5f280a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5f280a2b80 0x7f5f280a5040 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c005fd0 tx=0x7f5f3c00b590 comp rx=0 tx=0).stop 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 msgr2=0x7f5f4019ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4019ec50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5f3802ae70 tx=0x7f5f3802f5b0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 shutdown_connections 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5f280a2b80 0x7f5f280a5040 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f4010beb0 0x7f5f4019ec50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f40103430 0x7f5f4019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 --2- 172.21.15.64:0/2374823648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f40102b10 0x7f5f4019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 >> 172.21.15.64:0/2374823648 conn(0x7f5f400fc4d0 msgr2=0x7f5f40101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 shutdown_connections 2024-08-19T01:30:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:16.475+0000 7f5f46ee8640 1 -- 172.21.15.64:0/2374823648 wait complete. 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: Deploying daemon mds.d on smithi064 2024-08-19T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.64:6834/1183738942,v1:172.21.15.64:6835/1183738942] up:boot 2024-08-19T01:30:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: fsmap 3 up:standby 2024-08-19T01:30:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-08-19T01:30:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:16 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2374823648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:17.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:17.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: Deploying daemon mds.d on smithi064 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.64:6834/1183738942,v1:172.21.15.64:6835/1183738942] up:boot 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: fsmap 3 up:standby 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-08-19T01:30:17.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:16 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2374823648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: Deploying daemon mds.d on smithi064 2024-08-19T01:30:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.64:6834/1183738942,v1:172.21.15.64:6835/1183738942] up:boot 2024-08-19T01:30:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: fsmap 3 up:standby 2024-08-19T01:30:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-08-19T01:30:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:16 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2374823648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:18.733 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:18 smithi064 systemd[1]: Starting Ceph mds.d for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:18 smithi064 ceph-mon[18515]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:18.985 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:18 smithi064 podman[55371]: 2024-08-19 01:30:18.634859298 +0000 UTC m=+0.017267472 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:18.986 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:18 smithi064 podman[55371]: 2024-08-19 01:30:18.747563866 +0000 UTC m=+0.129972023 container create 41b5957e387d679190c72768a51bccc1c366f879245e01baa3c27288799b8adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-08-19T01:30:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:18 smithi123 ceph-mon[21829]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:19.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:18 smithi161 ceph-mon[21757]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:19.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:19.458 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:19.485 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:19 smithi064 podman[55371]: 2024-08-19 01:30:19.002531678 +0000 UTC m=+0.384939837 container init 41b5957e387d679190c72768a51bccc1c366f879245e01baa3c27288799b8adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T01:30:19.485 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:19 smithi064 podman[55371]: 2024-08-19 01:30:19.00526458 +0000 UTC m=+0.387672747 container start 41b5957e387d679190c72768a51bccc1c366f879245e01baa3c27288799b8adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-08-19T01:30:19.485 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:19 smithi064 ceph-mds[55390]: -- 172.21.15.64:0/3350135562 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55663b4cd860 con 0x55663b4aa800 2024-08-19T01:30:19.485 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:19 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d[55386]: starting mds.d at 2024-08-19T01:30:19.485 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:19 smithi064 bash[55371]: 41b5957e387d679190c72768a51bccc1c366f879245e01baa3c27288799b8adf 2024-08-19T01:30:19.485 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:30:19 smithi064 systemd[1]: Started Ceph mds.d for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:19.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 -- 172.21.15.64:0/2468362021 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b81078b0 msgr2=0x7f85b8107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:19.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/2468362021 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b81078b0 0x7f85b8107c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009930 tx=0x7f85b402a420 comp rx=0 tx=0).stop 2024-08-19T01:30:19.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 -- 172.21.15.64:0/2468362021 shutdown_connections 2024-08-19T01:30:19.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/2468362021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b8102250 0x7f85b810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:19.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/2468362021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85b81018b0 0x7f85b8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:19.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/2468362021 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b81078b0 0x7f85b8107c90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 -- 172.21.15.64:0/2468362021 >> 172.21.15.64:0/2468362021 conn(0x7f85b80fd5f0 msgr2=0x7f85b80ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 -- 172.21.15.64:0/2468362021 shutdown_connections 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.921+0000 7f85bfb8a640 1 -- 172.21.15.64:0/2468362021 wait complete. 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 Processor -- start 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 -- start start 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85b81018b0 0x7f85b8198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b8102250 0x7f85b8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b81078b0 0x7f85b819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f85b8111cf0 con 0x7f85b8102250 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f85b8111b70 con 0x7f85b81018b0 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bfb8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f85b8111e70 con 0x7f85b81078b0 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bd125640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b8102250 0x7f85b8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:19.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bd125640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b8102250 0x7f85b8198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52182/0 (socket says 172.21.15.64:52182) 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bd125640 1 -- 172.21.15.64:0/3994383643 learned_addr learned my addr 172.21.15.64:0/3994383643 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85bd926640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85b81018b0 0x7f85b8198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.922+0000 7f85be127640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b81078b0 0x7f85b819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bd125640 1 -- 172.21.15.64:0/3994383643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b81078b0 msgr2=0x7f85b819cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bd125640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b81078b0 0x7f85b819cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bd125640 1 -- 172.21.15.64:0/3994383643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85b81018b0 msgr2=0x7f85b8198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bd125640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85b81018b0 0x7f85b8198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bd125640 1 -- 172.21.15.64:0/3994383643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b819d1e0 con 0x7f85b8102250 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bd926640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85b81018b0 0x7f85b8198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bd125640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b8102250 0x7f85b8198770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00b4a0 tx=0x7f85ac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85aaffd640 1 -- 172.21.15.64:0/3994383643 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac014070 con 0x7f85b8102250 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85aaffd640 1 -- 172.21.15.64:0/3994383643 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85ac010af0 con 0x7f85b8102250 2024-08-19T01:30:19.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85aaffd640 1 -- 172.21.15.64:0/3994383643 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac017d10 con 0x7f85b8102250 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b819d4d0 con 0x7f85b8102250 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.923+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b81a4db0 con 0x7f85b8102250 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.924+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b8101d10 con 0x7f85b8102250 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.926+0000 7f85aaffd640 1 -- 172.21.15.64:0/3994383643 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f85ac017070 con 0x7f85b8102250 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.926+0000 7f85aaffd640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85a40a2c00 0x7f85a40a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.927+0000 7f85bd926640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85a40a2c00 0x7f85a40a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.927+0000 7f85aaffd640 1 -- 172.21.15.64:0/3994383643 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f85ac0ce2b0 con 0x7f85b8102250 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.927+0000 7f85bd926640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85a40a2c00 0x7f85a40a50c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f85b4000c00 tx=0x7f85b4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:19.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:19.930+0000 7f85aaffd640 1 -- 172.21.15.64:0/3994383643 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f85ac09b6b0 con 0x7f85b8102250 2024-08-19T01:30:20.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.140+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85b8199590 con 0x7f85b8102250 2024-08-19T01:30:20.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.141+0000 7f85aaffd640 1 -- 172.21.15.64:0/3994383643 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7f85b8199590 con 0x7f85b8102250 2024-08-19T01:30:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:20.143 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"btime":"2024-08-19T01:30:19:779350+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-08-19T01:30:20.143 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 5 2024-08-19T01:30:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85a40a2c00 msgr2=0x7f85a40a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85a40a2c00 0x7f85a40a50c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f85b4000c00 tx=0x7f85b4005a20 comp rx=0 tx=0).stop 2024-08-19T01:30:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b8102250 msgr2=0x7f85b8198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b8102250 0x7f85b8198770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00b4a0 tx=0x7f85ac00b960 comp rx=0 tx=0).stop 2024-08-19T01:30:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 shutdown_connections 2024-08-19T01:30:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85a40a2c00 0x7f85a40a50c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b81078b0 0x7f85b819cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.143+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85b8102250 0x7f85b8198770 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.144+0000 7f85bfb8a640 1 --2- 172.21.15.64:0/3994383643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85b81018b0 0x7f85b8198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.144+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 >> 172.21.15.64:0/3994383643 conn(0x7f85b80fd5f0 msgr2=0x7f85b80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.144+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 shutdown_connections 2024-08-19T01:30:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:20.144+0000 7f85bfb8a640 1 -- 172.21.15.64:0/3994383643 wait complete. 2024-08-19T01:30:20.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: Deploying daemon mds.e on smithi123 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.64:6836/404621970,v1:172.21.15.64:6837/404621970] up:boot 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: fsmap 4 up:standby 2024-08-19T01:30:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:20 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-08-19T01:30:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: Deploying daemon mds.e on smithi123 2024-08-19T01:30:20.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.64:6836/404621970,v1:172.21.15.64:6837/404621970] up:boot 2024-08-19T01:30:20.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: fsmap 4 up:standby 2024-08-19T01:30:20.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:20 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: Deploying daemon mds.e on smithi123 2024-08-19T01:30:20.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.64:6836/404621970,v1:172.21.15.64:6837/404621970] up:boot 2024-08-19T01:30:20.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: fsmap 4 up:standby 2024-08-19T01:30:20.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:20 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-08-19T01:30:21.145 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:21 smithi123 ceph-mon[21829]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:21.145 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:21 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3994383643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:21.416 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 systemd[1]: Starting Ceph mds.e for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:21.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:21 smithi161 ceph-mon[21757]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:21.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:21 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3994383643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:21.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:21 smithi064 ceph-mon[18515]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:21.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:21 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3994383643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:21.735 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 podman[51112]: 2024-08-19 01:30:21.363127801 +0000 UTC m=+0.017297529 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:21.735 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 podman[51112]: 2024-08-19 01:30:21.480555138 +0000 UTC m=+0.134724844 container create afb9d2fa33f82c612be9ac103e10cf4962a795d6eb799b4f4e908b69271647e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-19T01:30:21.735 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 podman[51112]: 2024-08-19 01:30:21.708187615 +0000 UTC m=+0.362357325 container init afb9d2fa33f82c612be9ac103e10cf4962a795d6eb799b4f4e908b69271647e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-08-19T01:30:21.735 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 podman[51112]: 2024-08-19 01:30:21.711099026 +0000 UTC m=+0.365268733 container start afb9d2fa33f82c612be9ac103e10cf4962a795d6eb799b4f4e908b69271647e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-08-19T01:30:22.166 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 ceph-mds[51131]: -- 172.21.15.123:0/3441589649 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56227e37b860 con 0x56227e359000 2024-08-19T01:30:22.166 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e[51127]: starting mds.e at 2024-08-19T01:30:22.166 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 bash[51112]: afb9d2fa33f82c612be9ac103e10cf4962a795d6eb799b4f4e908b69271647e0 2024-08-19T01:30:22.166 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:30:21 smithi123 systemd[1]: Started Ceph mds.e for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:22.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:22.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: Deploying daemon mds.f on smithi161 2024-08-19T01:30:22.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6834/2709386676,v1:172.21.15.123:6835/2709386676] up:boot 2024-08-19T01:30:22.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: fsmap 5 up:standby 2024-08-19T01:30:22.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:22 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:30:23.052 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: Deploying daemon mds.f on smithi161 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6834/2709386676,v1:172.21.15.123:6835/2709386676] up:boot 2024-08-19T01:30:23.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: fsmap 5 up:standby 2024-08-19T01:30:23.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:22 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:30:23.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:23.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:23.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: Deploying daemon mds.f on smithi161 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6834/2709386676,v1:172.21.15.123:6835/2709386676] up:boot 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: fsmap 5 up:standby 2024-08-19T01:30:23.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:22 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:30:23.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 -- 172.21.15.64:0/2946559938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c06a0a0 msgr2=0x7f7f7c1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:23.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 --2- 172.21.15.64:0/2946559938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c06a0a0 0x7f7f7c1006f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009ca0 tx=0x7f7f7002a8c0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 -- 172.21.15.64:0/2946559938 shutdown_connections 2024-08-19T01:30:23.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 --2- 172.21.15.64:0/2946559938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f7c104bb0 0x7f7f7c105060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 --2- 172.21.15.64:0/2946559938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c06a0a0 0x7f7f7c1006f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 --2- 172.21.15.64:0/2946559938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f7c069c30 0x7f7f7c1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 -- 172.21.15.64:0/2946559938 >> 172.21.15.64:0/2946559938 conn(0x7f7f7c076f40 msgr2=0x7f7f7c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.538+0000 7f7f84069640 1 -- 172.21.15.64:0/2946559938 shutdown_connections 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 -- 172.21.15.64:0/2946559938 wait complete. 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 Processor -- start 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 -- start start 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f7c069c30 0x7f7f7c102880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f7c06a0a0 0x7f7f7c102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c104bb0 0x7f7f7c103300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c110be0 con 0x7f7f7c104bb0 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c110a60 con 0x7f7f7c069c30 2024-08-19T01:30:23.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f84069640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c110d60 con 0x7f7f7c06a0a0 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f82606640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c104bb0 0x7f7f7c103300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f82606640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c104bb0 0x7f7f7c103300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52200/0 (socket says 172.21.15.64:52200) 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.539+0000 7f7f82606640 1 -- 172.21.15.64:0/3531377461 learned_addr learned my addr 172.21.15.64:0/3531377461 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f81604640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f7c06a0a0 0x7f7f7c102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f81e05640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f7c069c30 0x7f7f7c102880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f82606640 1 -- 172.21.15.64:0/3531377461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f7c06a0a0 msgr2=0x7f7f7c102dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f82606640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f7c06a0a0 0x7f7f7c102dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f82606640 1 -- 172.21.15.64:0/3531377461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f7c069c30 msgr2=0x7f7f7c102880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f82606640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f7c069c30 0x7f7f7c102880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f82606640 1 -- 172.21.15.64:0/3531377461 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c1a66b0 con 0x7f7f7c104bb0 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f82606640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c104bb0 0x7f7f7c103300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400d950 tx=0x7f7f7400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f7affd640 1 -- 172.21.15.64:0/3531377461 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74017030 con 0x7f7f7c104bb0 2024-08-19T01:30:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f7affd640 1 -- 172.21.15.64:0/3531377461 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f74004020 con 0x7f7f7c104bb0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7c1a69a0 con 0x7f7f7c104bb0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f7affd640 1 -- 172.21.15.64:0/3531377461 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74003c00 con 0x7f7f7c104bb0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.540+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7c1a6e60 con 0x7f7f7c104bb0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.541+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7c105060 con 0x7f7f7c104bb0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.542+0000 7f7f7affd640 1 -- 172.21.15.64:0/3531377461 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7f7401a070 con 0x7f7f7c104bb0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.543+0000 7f7f7affd640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7f640a2b80 0x7f7f640a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.543+0000 7f7f7affd640 1 -- 172.21.15.64:0/3531377461 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f7f740ce3f0 con 0x7f7f7c104bb0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.543+0000 7f7f81e05640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7f640a2b80 0x7f7f640a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.543+0000 7f7f81e05640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7f640a2b80 0x7f7f640a5040 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009b60 tx=0x7f7f6c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:23.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.547+0000 7f7f7affd640 1 -- 172.21.15.64:0/3531377461 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f7f74021070 con 0x7f7f7c104bb0 2024-08-19T01:30:23.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.760+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7c1a70f0 con 0x7f7f7c104bb0 2024-08-19T01:30:23.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.760+0000 7f7f7affd640 1 -- 172.21.15.64:0/3531377461 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4738 (secure 0 0 0) 0x7f7f7409b7f0 con 0x7f7f7c104bb0 2024-08-19T01:30:23.763 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:23.763 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":6,"btime":"2024-08-19T01:30:22:144521+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-08-19T01:30:23.763 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 6 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7f640a2b80 msgr2=0x7f7f640a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7f640a2b80 0x7f7f640a5040 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009b60 tx=0x7f7f6c005f90 comp rx=0 tx=0).stop 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c104bb0 msgr2=0x7f7f7c103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c104bb0 0x7f7f7c103300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400d950 tx=0x7f7f7400de10 comp rx=0 tx=0).stop 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 shutdown_connections 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7f640a2b80 0x7f7f640a5040 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f7c104bb0 0x7f7f7c103300 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f7c06a0a0 0x7f7f7c102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 --2- 172.21.15.64:0/3531377461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f7c069c30 0x7f7f7c102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:23.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 >> 172.21.15.64:0/3531377461 conn(0x7f7f7c076f40 msgr2=0x7f7f7c108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:23.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.762+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 shutdown_connections 2024-08-19T01:30:23.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:23.763+0000 7f7f84069640 1 -- 172.21.15.64:0/3531377461 wait complete. 2024-08-19T01:30:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:23 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3531377461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:24.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:23 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3531377461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:24.200 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 systemd[1]: Starting Ceph mds.f for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:24.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:23 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3531377461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:24.529 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 podman[51631]: 2024-08-19 01:30:24.177383627 +0000 UTC m=+0.017296353 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:24.529 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 podman[51631]: 2024-08-19 01:30:24.29280352 +0000 UTC m=+0.132716283 container create 3a9cfe8416775d08a129e94658cb10da919b1c72476d981d73bacedb5a72eb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True) 2024-08-19T01:30:24.529 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 podman[51631]: 2024-08-19 01:30:24.528496263 +0000 UTC m=+0.368408997 container init 3a9cfe8416775d08a129e94658cb10da919b1c72476d981d73bacedb5a72eb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-08-19T01:30:24.877 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 podman[51631]: 2024-08-19 01:30:24.531629549 +0000 UTC m=+0.371542268 container start 3a9cfe8416775d08a129e94658cb10da919b1c72476d981d73bacedb5a72eb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-08-19T01:30:24.877 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mds[51650]: -- 172.21.15.161:0/1735778585 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cab4683860 con 0x55cab4660800 2024-08-19T01:30:24.877 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f[51646]: starting mds.f at 2024-08-19T01:30:24.877 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 bash[51631]: 3a9cfe8416775d08a129e94658cb10da919b1c72476d981d73bacedb5a72eb35 2024-08-19T01:30:24.877 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:30:24 smithi161 systemd[1]: Started Ceph mds.f for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:25.129 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mon[21757]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:25.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:25.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:25.130 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:24 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:25.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:24 smithi123 ceph-mon[21829]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:25.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:24 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:25.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:24 smithi064 ceph-mon[18515]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:25.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:24 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:25 smithi123 ceph-mon[21829]: Deploying daemon mds.c on smithi161 2024-08-19T01:30:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:25 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6834/3530643584,v1:172.21.15.161:6835/3530643584] up:boot 2024-08-19T01:30:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:25 smithi123 ceph-mon[21829]: fsmap 6 up:standby 2024-08-19T01:30:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:25 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:30:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:25 smithi161 ceph-mon[21757]: Deploying daemon mds.c on smithi161 2024-08-19T01:30:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:25 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6834/3530643584,v1:172.21.15.161:6835/3530643584] up:boot 2024-08-19T01:30:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:25 smithi161 ceph-mon[21757]: fsmap 6 up:standby 2024-08-19T01:30:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:25 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:30:26.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:25 smithi064 ceph-mon[18515]: Deploying daemon mds.c on smithi161 2024-08-19T01:30:26.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:25 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6834/3530643584,v1:172.21.15.161:6835/3530643584] up:boot 2024-08-19T01:30:26.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:25 smithi064 ceph-mon[18515]: fsmap 6 up:standby 2024-08-19T01:30:26.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:25 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:30:26.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:26.770 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:26 smithi161 systemd[1]: Starting Ceph mds.c for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:26.813 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:27.095 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:26 smithi161 ceph-mon[21757]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:27.095 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:26 smithi161 podman[51934]: 2024-08-19 01:30:26.838341886 +0000 UTC m=+0.017379212 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:27.095 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:26 smithi161 podman[51934]: 2024-08-19 01:30:26.972769544 +0000 UTC m=+0.151806859 container create 1e98d414f9a496b881f72bda6fe80c9292dcd34c33ff4ba73e8c80f6f1df37e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-08-19T01:30:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:26 smithi123 ceph-mon[21829]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:27.216 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:26 smithi064 ceph-mon[18515]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:27.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 -- 172.21.15.64:0/1864710287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 msgr2=0x7f2f00107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:27.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 --2- 172.21.15.64:0/1864710287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f00107c90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009930 tx=0x7f2efc02a420 comp rx=0 tx=0).stop 2024-08-19T01:30:27.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 -- 172.21.15.64:0/1864710287 shutdown_connections 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 --2- 172.21.15.64:0/1864710287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f00102250 0x7f2f0010e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 --2- 172.21.15.64:0/1864710287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f001018b0 0x7f2f00101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 --2- 172.21.15.64:0/1864710287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f00107c90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 -- 172.21.15.64:0/1864710287 >> 172.21.15.64:0/1864710287 conn(0x7f2f000fd5f0 msgr2=0x7f2f000ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 -- 172.21.15.64:0/1864710287 shutdown_connections 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.295+0000 7f2f084a7640 1 -- 172.21.15.64:0/1864710287 wait complete. 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 Processor -- start 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 -- start start 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f001018b0 0x7f2f001981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:27.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f00102250 0x7f2f00198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f0019cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2f00111c70 con 0x7f2f001078b0 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2f00111af0 con 0x7f2f001018b0 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f084a7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f00111df0 con 0x7f2f00102250 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f06a44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f0019cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f06a44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f0019cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52232/0 (socket says 172.21.15.64:52232) 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f06a44640 1 -- 172.21.15.64:0/3646034750 learned_addr learned my addr 172.21.15.64:0/3646034750 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f05a42640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f00102250 0x7f2f00198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.296+0000 7f2f06243640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f001018b0 0x7f2f001981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f06a44640 1 -- 172.21.15.64:0/3646034750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f00102250 msgr2=0x7f2f00198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f06a44640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f00102250 0x7f2f00198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f06a44640 1 -- 172.21.15.64:0/3646034750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f001018b0 msgr2=0x7f2f001981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:27.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f06a44640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f001018b0 0x7f2f001981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f06a44640 1 -- 172.21.15.64:0/3646034750 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0019d1a0 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f06a44640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f0019cac0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800eb10 tx=0x7f2ef800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2ef77fe640 1 -- 172.21.15.64:0/3646034750 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8004b20 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2ef77fe640 1 -- 172.21.15.64:0/3646034750 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ef8004cc0 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f0019d490 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2ef77fe640 1 -- 172.21.15.64:0/3646034750 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8010760 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.297+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f001a4d70 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.298+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f00101d10 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.299+0000 7f2ef77fe640 1 -- 172.21.15.64:0/3646034750 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2ef8005020 con 0x7f2f001078b0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.300+0000 7f2ef77fe640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2eec0a2c00 0x7f2eec0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.300+0000 7f2f06243640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2eec0a2c00 0x7f2eec0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:27.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.300+0000 7f2ef77fe640 1 -- 172.21.15.64:0/3646034750 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f2ef80cf030 con 0x7f2f001078b0 2024-08-19T01:30:27.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.300+0000 7f2f06243640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2eec0a2c00 0x7f2eec0a50c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009900 tx=0x7f2efc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:27.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.304+0000 7f2ef77fe640 1 -- 172.21.15.64:0/3646034750 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f2ef809c430 con 0x7f2f001078b0 2024-08-19T01:30:27.449 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:27 smithi161 podman[51934]: 2024-08-19 01:30:27.226192611 +0000 UTC m=+0.405229944 container init 1e98d414f9a496b881f72bda6fe80c9292dcd34c33ff4ba73e8c80f6f1df37e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:30:27.449 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:27 smithi161 podman[51934]: 2024-08-19 01:30:27.229168435 +0000 UTC m=+0.408205755 container start 1e98d414f9a496b881f72bda6fe80c9292dcd34c33ff4ba73e8c80f6f1df37e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-08-19T01:30:27.449 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:27 smithi161 ceph-mds[51963]: -- 172.21.15.161:0/2274039112 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eab1729860 con 0x55eab1707400 2024-08-19T01:30:27.449 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c[51957]: starting mds.c at 2024-08-19T01:30:27.449 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:27 smithi161 bash[51934]: 1e98d414f9a496b881f72bda6fe80c9292dcd34c33ff4ba73e8c80f6f1df37e0 2024-08-19T01:30:27.450 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:30:27 smithi161 systemd[1]: Started Ceph mds.c for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:27.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.519+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f0002d5e0 con 0x7f2f001078b0 2024-08-19T01:30:27.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.519+0000 7f2ef77fe640 1 -- 172.21.15.64:0/3646034750 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5566 (secure 0 0 0) 0x7f2ef809bd30 con 0x7f2f001078b0 2024-08-19T01:30:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:27.521 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":7,"btime":"2024-08-19T01:30:24:873165+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-08-19T01:30:27.521 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 7 2024-08-19T01:30:27.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2eec0a2c00 msgr2=0x7f2eec0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:27.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2eec0a2c00 0x7f2eec0a50c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009900 tx=0x7f2efc0058d0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 msgr2=0x7f2f0019cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:27.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f0019cac0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800eb10 tx=0x7f2ef800c2d0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 shutdown_connections 2024-08-19T01:30:27.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2eec0a2c00 0x7f2eec0a50c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f001078b0 0x7f2f0019cac0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f00102250 0x7f2f00198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 --2- 172.21.15.64:0/3646034750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f001018b0 0x7f2f001981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:27.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.521+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 >> 172.21.15.64:0/3646034750 conn(0x7f2f000fd5f0 msgr2=0x7f2f000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:27.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.522+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 shutdown_connections 2024-08-19T01:30:27.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:27.522+0000 7f2f084a7640 1 -- 172.21.15.64:0/3646034750 wait complete. 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: Deploying daemon mds.h on smithi123 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3646034750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:28.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6836/1326350329,v1:172.21.15.161:6837/1326350329] up:boot 2024-08-19T01:30:28.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: fsmap 7 up:standby 2024-08-19T01:30:28.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:28 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:30:28.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:28.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:28.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:28.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: Deploying daemon mds.h on smithi123 2024-08-19T01:30:28.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3646034750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:28.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6836/1326350329,v1:172.21.15.161:6837/1326350329] up:boot 2024-08-19T01:30:28.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: fsmap 7 up:standby 2024-08-19T01:30:28.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:28 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: Deploying daemon mds.h on smithi123 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3646034750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6836/1326350329,v1:172.21.15.161:6837/1326350329] up:boot 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: fsmap 7 up:standby 2024-08-19T01:30:28.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:28 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:30:29.604 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:29 smithi123 ceph-mon[21829]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:29.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:29 smithi161 ceph-mon[21757]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:29.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:29 smithi064 ceph-mon[18515]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:30.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:30.165 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:30:29 smithi123 systemd[1]: Started Ceph mds.h for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:30.406 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:30.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 -- 172.21.15.64:0/2906037405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 msgr2=0x7f0e1c100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:30.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 --2- 172.21.15.64:0/2906037405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c100f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009900 tx=0x7f0e1002a480 comp rx=0 tx=0).stop 2024-08-19T01:30:30.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 -- 172.21.15.64:0/2906037405 shutdown_connections 2024-08-19T01:30:30.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 --2- 172.21.15.64:0/2906037405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e1c102c20 0x7f0e1c10d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:30.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 --2- 172.21.15.64:0/2906037405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e1c1022a0 0x7f0e1c1026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:30.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 --2- 172.21.15.64:0/2906037405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c100f30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:30.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 -- 172.21.15.64:0/2906037405 >> 172.21.15.64:0/2906037405 conn(0x7f0e1c0fc500 msgr2=0x7f0e1c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:30.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 -- 172.21.15.64:0/2906037405 shutdown_connections 2024-08-19T01:30:30.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 -- 172.21.15.64:0/2906037405 wait complete. 2024-08-19T01:30:30.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.907+0000 7f0e22b57640 1 Processor -- start 2024-08-19T01:30:30.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e22b57640 1 -- start start 2024-08-19T01:30:30.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e22b57640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:30.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e22b57640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e1c1022a0 0x7f0e1c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:30.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e22b57640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e1c102c20 0x7f0e1c19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:30.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e22b57640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1c110cb0 con 0x7f0e1c100b50 2024-08-19T01:30:30.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e22b57640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1c110b30 con 0x7f0e1c102c20 2024-08-19T01:30:30.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e22b57640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1c110e30 con 0x7f0e1c1022a0 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c1981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42342/0 (socket says 172.21.15.64:42342) 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 -- 172.21.15.64:0/1440963002 learned_addr learned my addr 172.21.15.64:0/1440963002 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e210f4640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e1c102c20 0x7f0e1c19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e1bfff640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e1c1022a0 0x7f0e1c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 -- 172.21.15.64:0/1440963002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e1c1022a0 msgr2=0x7f0e1c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e1c1022a0 0x7f0e1c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 -- 172.21.15.64:0/1440963002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e1c102c20 msgr2=0x7f0e1c19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e1c102c20 0x7f0e1c19ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e208f3640 1 -- 172.21.15.64:0/1440963002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1c19d150 con 0x7f0e1c100b50 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e210f4640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e1c102c20 0x7f0e1c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.908+0000 7f0e1bfff640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e1c1022a0 0x7f0e1c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:30:30.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.909+0000 7f0e208f3640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c1981a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006010 tx=0x7f0e10003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:30.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.909+0000 7f0e19ffb640 1 -- 172.21.15.64:0/1440963002 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1003f070 con 0x7f0e1c100b50 2024-08-19T01:30:30.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.909+0000 7f0e19ffb640 1 -- 172.21.15.64:0/1440963002 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e10031c30 con 0x7f0e1c100b50 2024-08-19T01:30:30.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.909+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1c19d3e0 con 0x7f0e1c100b50 2024-08-19T01:30:30.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.909+0000 7f0e19ffb640 1 -- 172.21.15.64:0/1440963002 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e10035030 con 0x7f0e1c100b50 2024-08-19T01:30:30.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.909+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e1c1a4d20 con 0x7f0e1c100b50 2024-08-19T01:30:30.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.910+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e1c100f30 con 0x7f0e1c100b50 2024-08-19T01:30:30.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.911+0000 7f0e19ffb640 1 -- 172.21.15.64:0/1440963002 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0e10042070 con 0x7f0e1c100b50 2024-08-19T01:30:30.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.912+0000 7f0e19ffb640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f0e0c0a2b80 0x7f0e0c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:30.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.912+0000 7f0e1bfff640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f0e0c0a2b80 0x7f0e0c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:30.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.912+0000 7f0e19ffb640 1 -- 172.21.15.64:0/1440963002 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f0e100ef850 con 0x7f0e1c100b50 2024-08-19T01:30:30.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.913+0000 7f0e1bfff640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f0e0c0a2b80 0x7f0e0c0a5040 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c06ad80 tx=0x7f0e0800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:30.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:30.916+0000 7f0e19ffb640 1 -- 172.21.15.64:0/1440963002 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f0e10031de0 con 0x7f0e1c100b50 2024-08-19T01:30:31.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.128+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e1c1a4ef0 con 0x7f0e1c100b50 2024-08-19T01:30:31.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.128+0000 7f0e19ffb640 1 -- 172.21.15.64:0/1440963002 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7222 (secure 0 0 0) 0x7f0e100bc620 con 0x7f0e1c100b50 2024-08-19T01:30:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:31.130 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":9,"btime":"2024-08-19T01:30:30:388735+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24421,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-08-19T01:30:31.130 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 9 2024-08-19T01:30:31.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f0e0c0a2b80 msgr2=0x7f0e0c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:31.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f0e0c0a2b80 0x7f0e0c0a5040 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c06ad80 tx=0x7f0e0800b390 comp rx=0 tx=0).stop 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 msgr2=0x7f0e1c1981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c1981a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006010 tx=0x7f0e10003c10 comp rx=0 tx=0).stop 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 shutdown_connections 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f0e0c0a2b80 0x7f0e0c0a5040 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e1c102c20 0x7f0e1c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e1c1022a0 0x7f0e1c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 --2- 172.21.15.64:0/1440963002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e1c100b50 0x7f0e1c1981a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 >> 172.21.15.64:0/1440963002 conn(0x7f0e1c0fc500 msgr2=0x7f0e1c105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 shutdown_connections 2024-08-19T01:30:31.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:31.131+0000 7f0e22b57640 1 -- 172.21.15.64:0/1440963002 wait complete. 2024-08-19T01:30:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:31.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:31.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: Deploying daemon mds.i on smithi161 2024-08-19T01:30:31.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] up:boot 2024-08-19T01:30:31.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: fsmap 8 up:standby 2024-08-19T01:30:31.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:31 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:31.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: Deploying daemon mds.i on smithi161 2024-08-19T01:30:31.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] up:boot 2024-08-19T01:30:31.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: fsmap 8 up:standby 2024-08-19T01:30:31.417 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:31 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:30:31.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: Deploying daemon mds.i on smithi161 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] up:boot 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: fsmap 8 up:standby 2024-08-19T01:30:31.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:31 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:30:32.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:32 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1440963002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:32.203 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 systemd[1]: Starting Ceph mds.i for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:32 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1440963002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:32.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:32 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1440963002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:32.534 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 podman[52246]: 2024-08-19 01:30:32.176160108 +0000 UTC m=+0.017343693 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:32.535 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 podman[52246]: 2024-08-19 01:30:32.288757764 +0000 UTC m=+0.129941342 container create 8a581b31383a95413781cb77fd1ea3cb78a53bbf310016e09a3b6c11d44f738e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-08-19T01:30:32.535 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 podman[52246]: 2024-08-19 01:30:32.507944102 +0000 UTC m=+0.349127688 container init 8a581b31383a95413781cb77fd1ea3cb78a53bbf310016e09a3b6c11d44f738e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-08-19T01:30:32.535 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 podman[52246]: 2024-08-19 01:30:32.510653473 +0000 UTC m=+0.351837054 container start 8a581b31383a95413781cb77fd1ea3cb78a53bbf310016e09a3b6c11d44f738e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-19T01:30:32.949 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 ceph-mds[52265]: -- 172.21.15.161:0/2986400428 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55845655f860 con 0x55845653c800 2024-08-19T01:30:32.949 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i[52261]: starting mds.i at 2024-08-19T01:30:32.949 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 bash[52246]: 8a581b31383a95413781cb77fd1ea3cb78a53bbf310016e09a3b6c11d44f738e 2024-08-19T01:30:32.949 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:30:32 smithi161 systemd[1]: Started Ceph mds.i for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:33.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:33 smithi064 ceph-mon[18515]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:33.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:33.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:33.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:33.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:33 smithi123 ceph-mon[21829]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:33.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:33.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:33.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:33 smithi161 ceph-mon[21757]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:33.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:33.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:34 smithi064 ceph-mon[18515]: Deploying daemon mds.g on smithi064 2024-08-19T01:30:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:34 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] up:boot 2024-08-19T01:30:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:34 smithi064 ceph-mon[18515]: fsmap 9 up:standby 2024-08-19T01:30:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:34 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:30:34.329 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:34.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:34 smithi123 ceph-mon[21829]: Deploying daemon mds.g on smithi064 2024-08-19T01:30:34.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:34 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] up:boot 2024-08-19T01:30:34.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:34 smithi123 ceph-mon[21829]: fsmap 9 up:standby 2024-08-19T01:30:34.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:34 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:30:34.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:34 smithi161 ceph-mon[21757]: Deploying daemon mds.g on smithi064 2024-08-19T01:30:34.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:34 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] up:boot 2024-08-19T01:30:34.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:34 smithi161 ceph-mon[21757]: fsmap 9 up:standby 2024-08-19T01:30:34.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:34 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:30:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 -- 172.21.15.64:0/2973474738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41007c0 msgr2=0x7f77b4100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 --2- 172.21.15.64:0/2973474738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41007c0 0x7f77b4100c20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f77a8009930 tx=0x7f77a802a420 comp rx=0 tx=0).stop 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 -- 172.21.15.64:0/2973474738 shutdown_connections 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 --2- 172.21.15.64:0/2973474738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77b4101160 0x7f77b410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 --2- 172.21.15.64:0/2973474738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41007c0 0x7f77b4100c20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 --2- 172.21.15.64:0/2973474738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77b41067c0 0x7f77b4106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 -- 172.21.15.64:0/2973474738 >> 172.21.15.64:0/2973474738 conn(0x7f77b40fc500 msgr2=0x7f77b40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 -- 172.21.15.64:0/2973474738 shutdown_connections 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.942+0000 7f77ba129640 1 -- 172.21.15.64:0/2973474738 wait complete. 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 Processor -- start 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 -- start start 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77b41007c0 0x7f77b4199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77b4101160 0x7f77b41995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41067c0 0x7f77b41a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f77b4110be0 con 0x7f77b41067c0 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f77b4110a60 con 0x7f77b4101160 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77ba129640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f77b4110d60 con 0x7f77b41007c0 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77b3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41067c0 0x7f77b41a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77b3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41067c0 0x7f77b41a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42360/0 (socket says 172.21.15.64:42360) 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.943+0000 7f77b3fff640 1 -- 172.21.15.64:0/1957753638 learned_addr learned my addr 172.21.15.64:0/1957753638 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b2ffd640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77b4101160 0x7f77b41995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b37fe640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77b41007c0 0x7f77b4199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b3fff640 1 -- 172.21.15.64:0/1957753638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77b41007c0 msgr2=0x7f77b4199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b3fff640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77b41007c0 0x7f77b4199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b3fff640 1 -- 172.21.15.64:0/1957753638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77b4101160 msgr2=0x7f77b41995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b3fff640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77b4101160 0x7f77b41995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b3fff640 1 -- 172.21.15.64:0/1957753638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b41a4630 con 0x7f77b41067c0 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b3fff640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41067c0 0x7f77b41a40f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00b970 tx=0x7f77ac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b0ff9640 1 -- 172.21.15.64:0/1957753638 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac019070 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b41a4800 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b0ff9640 1 -- 172.21.15.64:0/1957753638 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77ac00cb40 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77b0ff9640 1 -- 172.21.15.64:0/1957753638 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac007610 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.944+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b41a4d10 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.945+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b4100c20 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.946+0000 7f77b0ff9640 1 -- 172.21.15.64:0/1957753638 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f77ac01c070 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.947+0000 7f77b0ff9640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f779c0a2b80 0x7f779c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.947+0000 7f77b37fe640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f779c0a2b80 0x7f779c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.947+0000 7f77b0ff9640 1 -- 172.21.15.64:0/1957753638 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f77ac0ce140 con 0x7f77b41067c0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.947+0000 7f77b37fe640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f779c0a2b80 0x7f779c0a5040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f77a40058c0 tx=0x7f77a400a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:34.951+0000 7f77b0ff9640 1 -- 172.21.15.64:0/1957753638 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f77ac09aef0 con 0x7f77b41067c0 2024-08-19T01:30:35.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:35 smithi064 ceph-mon[18515]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:35.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.172+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77b4108d50 con 0x7f77b41067c0 2024-08-19T01:30:35.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.173+0000 7f77b0ff9640 1 -- 172.21.15.64:0/1957753638 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8052 (secure 0 0 0) 0x7f77ac09f1f0 con 0x7f77b41067c0 2024-08-19T01:30:35.174 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:35.174 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"btime":"2024-08-19T01:30:33:045636+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24421,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-08-19T01:30:35.175 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 10 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f779c0a2b80 msgr2=0x7f779c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f779c0a2b80 0x7f779c0a5040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f77a40058c0 tx=0x7f77a400a4c0 comp rx=0 tx=0).stop 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41067c0 msgr2=0x7f77b41a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41067c0 0x7f77b41a40f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00b970 tx=0x7f77ac00be30 comp rx=0 tx=0).stop 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 shutdown_connections 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f779c0a2b80 0x7f779c0a5040 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77b41067c0 0x7f77b41a40f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77b4101160 0x7f77b41995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 --2- 172.21.15.64:0/1957753638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77b41007c0 0x7f77b4199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 >> 172.21.15.64:0/1957753638 conn(0x7f77b40fc500 msgr2=0x7f77b40fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 shutdown_connections 2024-08-19T01:30:35.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:35.175+0000 7f77ba129640 1 -- 172.21.15.64:0/1957753638 wait complete. 2024-08-19T01:30:35.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:35 smithi123 ceph-mon[21829]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:35.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:35 smithi161 ceph-mon[21757]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:36.271 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:36 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1957753638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:36.271 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:30:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-g[56273]: starting mds.g at 2024-08-19T01:30:36.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:36 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1957753638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:36.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:36 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1957753638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:36.734 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:30:36 smithi064 bash[56245]: f5862aed040d9708f017cb23116a881afa85837aeab7eb7ce1bc7fb520c769b5 2024-08-19T01:30:36.735 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:30:36 smithi064 systemd[1]: Started Ceph mds.g for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:37.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:37.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:37.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:37.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:37.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: Deploying daemon mds.a on smithi064 2024-08-19T01:30:37.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.64:6838/2883191544,v1:172.21.15.64:6839/2883191544] up:boot 2024-08-19T01:30:37.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: fsmap 10 up:standby 2024-08-19T01:30:37.463 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:37 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: Deploying daemon mds.a on smithi064 2024-08-19T01:30:37.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.64:6838/2883191544,v1:172.21.15.64:6839/2883191544] up:boot 2024-08-19T01:30:37.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: fsmap 10 up:standby 2024-08-19T01:30:37.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:37 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-08-19T01:30:37.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:37.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:37.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:37.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:37.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:37.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: Deploying daemon mds.a on smithi064 2024-08-19T01:30:37.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.64:6838/2883191544,v1:172.21.15.64:6839/2883191544] up:boot 2024-08-19T01:30:37.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: fsmap 10 up:standby 2024-08-19T01:30:37.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:37 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-08-19T01:30:38.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:38.475 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 systemd[1]: Starting Ceph mds.a for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:38.735 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 podman[56580]: 2024-08-19 01:30:38.3763495 +0000 UTC m=+0.026662028 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:38.735 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 podman[56580]: 2024-08-19 01:30:38.494571192 +0000 UTC m=+0.144883758 container create dd537a28c24f2ae26e166b826c9090b35535f44a1a7cc048520ee6e33e5fc80e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:30:38.735 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 podman[56580]: 2024-08-19 01:30:38.717372651 +0000 UTC m=+0.367685193 container init dd537a28c24f2ae26e166b826c9090b35535f44a1a7cc048520ee6e33e5fc80e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:30:38.735 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 podman[56580]: 2024-08-19 01:30:38.720352953 +0000 UTC m=+0.370665477 container start dd537a28c24f2ae26e166b826c9090b35535f44a1a7cc048520ee6e33e5fc80e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:30:38.868 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:39.011 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 ceph-mds[56624]: -- 172.21.15.64:0/4267400304 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645dc81b860 con 0x5645dc7f8800 2024-08-19T01:30:39.011 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a[56609]: starting mds.a at 2024-08-19T01:30:39.011 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 bash[56580]: dd537a28c24f2ae26e166b826c9090b35535f44a1a7cc048520ee6e33e5fc80e 2024-08-19T01:30:39.011 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:30:38 smithi064 systemd[1]: Started Ceph mds.a for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:39.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 -- 172.21.15.64:0/1282121234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 msgr2=0x7fe8a010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:39.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 --2- 172.21.15.64:0/1282121234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a010d770 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe898009ca0 tx=0x7fe89802a950 comp rx=0 tx=0).stop 2024-08-19T01:30:39.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 -- 172.21.15.64:0/1282121234 shutdown_connections 2024-08-19T01:30:39.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 --2- 172.21.15.64:0/1282121234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a010d770 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 --2- 172.21.15.64:0/1282121234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a01022a0 0x7fe8a01026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 --2- 172.21.15.64:0/1282121234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8a0100b50 0x7fe8a0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 -- 172.21.15.64:0/1282121234 >> 172.21.15.64:0/1282121234 conn(0x7fe8a00fc500 msgr2=0x7fe8a00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 -- 172.21.15.64:0/1282121234 shutdown_connections 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.352+0000 7fe8a620e640 1 -- 172.21.15.64:0/1282121234 wait complete. 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 Processor -- start 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 -- start start 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8a0100b50 0x7fe8a019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a01022a0 0x7fe8a019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0111dd0 con 0x7fe8a0102c20 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0111c50 con 0x7fe8a0100b50 2024-08-19T01:30:39.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.353+0000 7fe8a620e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0111f50 con 0x7fe8a01022a0 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a019ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34462/0 (socket says 172.21.15.64:34462) 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 -- 172.21.15.64:0/4165446720 learned_addr learned my addr 172.21.15.64:0/4165446720 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89f7fe640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8a0100b50 0x7fe8a019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89effd640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a01022a0 0x7fe8a019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 -- 172.21.15.64:0/4165446720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a01022a0 msgr2=0x7fe8a019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a01022a0 0x7fe8a019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 -- 172.21.15.64:0/4165446720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8a0100b50 msgr2=0x7fe8a019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8a0100b50 0x7fe8a019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 -- 172.21.15.64:0/4165446720 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a019f3c0 con 0x7fe8a0102c20 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89ffff640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a019ece0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe898006010 tx=0x7fe89802f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89cff9640 1 -- 172.21.15.64:0/4165446720 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898034040 con 0x7fe8a0102c20 2024-08-19T01:30:39.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89cff9640 1 -- 172.21.15.64:0/4165446720 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe898003e00 con 0x7fe8a0102c20 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a019f650 con 0x7fe8a0102c20 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe89cff9640 1 -- 172.21.15.64:0/4165446720 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898045480 con 0x7fe8a0102c20 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.354+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a01a6f90 con 0x7fe8a0102c20 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.355+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a0100f30 con 0x7fe8a0102c20 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.356+0000 7fe89cff9640 1 -- 172.21.15.64:0/4165446720 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe898037020 con 0x7fe8a0102c20 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.357+0000 7fe89cff9640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8880a2c00 0x7fe8880a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.358+0000 7fe89f7fe640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8880a2c00 0x7fe8880a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.358+0000 7fe89cff9640 1 -- 172.21.15.64:0/4165446720 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fe8980efcc0 con 0x7fe8a0102c20 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.358+0000 7fe89f7fe640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8880a2c00 0x7fe8880a50c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe8900058c0 tx=0x7fe89000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:39.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.361+0000 7fe89cff9640 1 -- 172.21.15.64:0/4165446720 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe8980bd0b0 con 0x7fe8a0102c20 2024-08-19T01:30:39.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.571+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8a019b770 con 0x7fe8a0102c20 2024-08-19T01:30:39.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:39 smithi064 ceph-mon[18515]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:39.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:39.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:39.575 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:39 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:39.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.572+0000 7fe89cff9640 1 -- 172.21.15.64:0/4165446720 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9701 (secure 0 0 0) 0x7fe8a019b770 con 0x7fe8a0102c20 2024-08-19T01:30:39.576 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:39.576 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":12,"btime":"2024-08-19T01:30:39:321677+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24421,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-08-19T01:30:39.576 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 12 2024-08-19T01:30:39.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8880a2c00 msgr2=0x7fe8880a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:39.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8880a2c00 0x7fe8880a50c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe8900058c0 tx=0x7fe89000a2d0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 msgr2=0x7fe8a019ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:39.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a019ece0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe898006010 tx=0x7fe89802f6f0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 shutdown_connections 2024-08-19T01:30:39.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8880a2c00 0x7fe8880a50c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8a0102c20 0x7fe8a019ece0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a01022a0 0x7fe8a019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 --2- 172.21.15.64:0/4165446720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8a0100b50 0x7fe8a019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:39.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.574+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 >> 172.21.15.64:0/4165446720 conn(0x7fe8a00fc500 msgr2=0x7fe8a0106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:39.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.575+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 shutdown_connections 2024-08-19T01:30:39.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:39.575+0000 7fe8a620e640 1 -- 172.21.15.64:0/4165446720 wait complete. 2024-08-19T01:30:39.581 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:39 smithi123 ceph-mon[21829]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:39.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:39.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:39.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:39 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:39.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:39 smithi161 ceph-mon[21757]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:39.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:39.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-19T01:30:39.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-19T01:30:39.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:39 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:40.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:40 smithi123 ceph-mon[21829]: Deploying daemon mds.b on smithi123 2024-08-19T01:30:40.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:40 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.64:6840/799479773,v1:172.21.15.64:6841/799479773] up:boot 2024-08-19T01:30:40.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:40 smithi123 ceph-mon[21829]: fsmap 11 up:standby 2024-08-19T01:30:40.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:40 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-08-19T01:30:40.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:40 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4165446720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:40.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:40 smithi161 ceph-mon[21757]: Deploying daemon mds.b on smithi123 2024-08-19T01:30:40.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:40 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.64:6840/799479773,v1:172.21.15.64:6841/799479773] up:boot 2024-08-19T01:30:40.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:40 smithi161 ceph-mon[21757]: fsmap 11 up:standby 2024-08-19T01:30:40.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:40 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-08-19T01:30:40.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:40 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4165446720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:40.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:40 smithi064 ceph-mon[18515]: Deploying daemon mds.b on smithi123 2024-08-19T01:30:40.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:40 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.64:6840/799479773,v1:172.21.15.64:6841/799479773] up:boot 2024-08-19T01:30:40.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:40 smithi064 ceph-mon[18515]: fsmap 11 up:standby 2024-08-19T01:30:40.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:40 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-08-19T01:30:40.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:40 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4165446720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:41.149 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:40 smithi123 systemd[1]: Starting Ceph mds.b for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:30:41.412 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:41 smithi123 ceph-mon[21829]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:41.413 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 podman[51720]: 2024-08-19 01:30:41.050191069 +0000 UTC m=+0.017527132 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:30:41.413 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 podman[51720]: 2024-08-19 01:30:41.266798732 +0000 UTC m=+0.234134784 container create 532c3f8b51c467033319bf770c7810973cd4fc3a86c34aae7823f2486bd7b60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True) 2024-08-19T01:30:41.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:41 smithi161 ceph-mon[21757]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:41.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:41 smithi064 ceph-mon[18515]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:41.916 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 podman[51720]: 2024-08-19 01:30:41.51146689 +0000 UTC m=+0.478802943 container init 532c3f8b51c467033319bf770c7810973cd4fc3a86c34aae7823f2486bd7b60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:30:41.916 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 podman[51720]: 2024-08-19 01:30:41.514235996 +0000 UTC m=+0.481572049 container start 532c3f8b51c467033319bf770c7810973cd4fc3a86c34aae7823f2486bd7b60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731) 2024-08-19T01:30:41.916 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 ceph-mds[51739]: -- 172.21.15.123:0/1158094779 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ad1975860 con 0x564ad1953000 2024-08-19T01:30:41.916 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: starting mds.b at 2024-08-19T01:30:41.916 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 bash[51720]: 532c3f8b51c467033319bf770c7810973cd4fc3a86c34aae7823f2486bd7b60b 2024-08-19T01:30:41.916 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:41 smithi123 systemd[1]: Started Ceph mds.b for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:30:42.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:30:42.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:42.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:42.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] up:boot 2024-08-19T01:30:42.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: fsmap 12 up:standby 2024-08-19T01:30:42.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:42 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:30:42.889 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:42.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] up:boot 2024-08-19T01:30:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: fsmap 12 up:standby 2024-08-19T01:30:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:42 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:30:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:42.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:42.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:42.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] up:boot 2024-08-19T01:30:42.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: fsmap 12 up:standby 2024-08-19T01:30:42.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:42 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 -- 172.21.15.64:0/724645162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 msgr2=0x7fd3b4100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/724645162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4100c10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009930 tx=0x7fd3a802a420 comp rx=0 tx=0).stop 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 -- 172.21.15.64:0/724645162 shutdown_connections 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/724645162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b4101150 0x7fd3b410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/724645162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4100c10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/724645162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b41067b0 0x7fd3b4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 -- 172.21.15.64:0/724645162 >> 172.21.15.64:0/724645162 conn(0x7fd3b40fc4d0 msgr2=0x7fd3b40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 -- 172.21.15.64:0/724645162 shutdown_connections 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.372+0000 7fd3b98f7640 1 -- 172.21.15.64:0/724645162 wait complete. 2024-08-19T01:30:43.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 Processor -- start 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 -- start start 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b4101150 0x7fd3b41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b41067b0 0x7fd3b419b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b4110bb0 con 0x7fd3b41007b0 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b4110a30 con 0x7fd3b41067b0 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b98f7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b4110d30 con 0x7fd3b4101150 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34484/0 (socket says 172.21.15.64:34484) 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.373+0000 7fd3b2ffd640 1 -- 172.21.15.64:0/3285792988 learned_addr learned my addr 172.21.15.64:0/3285792988 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:43.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b27fc640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b4101150 0x7fd3b41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b37fe640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b41067b0 0x7fd3b419b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b2ffd640 1 -- 172.21.15.64:0/3285792988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b4101150 msgr2=0x7fd3b41975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b2ffd640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b4101150 0x7fd3b41975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b2ffd640 1 -- 172.21.15.64:0/3285792988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b41067b0 msgr2=0x7fd3b419b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b2ffd640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b41067b0 0x7fd3b419b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b2ffd640 1 -- 172.21.15.64:0/3285792988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b419c020 con 0x7fd3b41007b0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b2ffd640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4197070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400d950 tx=0x7fd3a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b88f5640 1 -- 172.21.15.64:0/3285792988 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4017030 con 0x7fd3b41007b0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b88f5640 1 -- 172.21.15.64:0/3285792988 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3a4004020 con 0x7fd3b41007b0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b419c310 con 0x7fd3b41007b0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b88f5640 1 -- 172.21.15.64:0/3285792988 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40047a0 con 0x7fd3b41007b0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.374+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b406a4c0 con 0x7fd3b41007b0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.375+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b4100c10 con 0x7fd3b41007b0 2024-08-19T01:30:43.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.377+0000 7fd3b88f5640 1 -- 172.21.15.64:0/3285792988 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd3a40041d0 con 0x7fd3b41007b0 2024-08-19T01:30:43.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.378+0000 7fd3b88f5640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd39c0a2c00 0x7fd39c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:43.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.378+0000 7fd3b27fc640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd39c0a2c00 0x7fd39c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:43.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.378+0000 7fd3b88f5640 1 -- 172.21.15.64:0/3285792988 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fd3a40cea70 con 0x7fd3b41007b0 2024-08-19T01:30:43.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.379+0000 7fd3b27fc640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd39c0a2c00 0x7fd39c0a50c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009900 tx=0x7fd3a80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:43.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.380+0000 7fd3b88f5640 1 -- 172.21.15.64:0/3285792988 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd3a409bef0 con 0x7fd3b41007b0 2024-08-19T01:30:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.636+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3b41983d0 con 0x7fd3b41007b0 2024-08-19T01:30:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.637+0000 7fd3b88f5640 1 -- 172.21.15.64:0/3285792988 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10530 (secure 0 0 0) 0x7fd3a409b7f0 con 0x7fd3b41007b0 2024-08-19T01:30:43.640 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:43.640 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":13,"btime":"2024-08-19T01:30:42:337499+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24421,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-08-19T01:30:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 13 2024-08-19T01:30:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd39c0a2c00 msgr2=0x7fd39c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd39c0a2c00 0x7fd39c0a50c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009900 tx=0x7fd3a80058d0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 msgr2=0x7fd3b4197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4197070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400d950 tx=0x7fd3a400de10 comp rx=0 tx=0).stop 2024-08-19T01:30:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 shutdown_connections 2024-08-19T01:30:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd39c0a2c00 0x7fd39c0a50c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b41067b0 0x7fd3b419b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b4101150 0x7fd3b41975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 --2- 172.21.15.64:0/3285792988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3b41007b0 0x7fd3b4197070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 >> 172.21.15.64:0/3285792988 conn(0x7fd3b40fc4d0 msgr2=0x7fd3b40fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.639+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 shutdown_connections 2024-08-19T01:30:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:43.640+0000 7fd3b98f7640 1 -- 172.21.15.64:0/3285792988 wait complete. 2024-08-19T01:30:44.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-08-19T01:30:44.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:44 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3285792988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:44.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:44 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:44 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3285792988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:44.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:44 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.678 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:44.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:44.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:44 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3285792988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:30:44.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:44 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 -- 172.21.15.64:0/4228176418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 msgr2=0x7f6a9c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/4228176418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c101d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009930 tx=0x7f6a9002a420 comp rx=0 tx=0).stop 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 -- 172.21.15.64:0/4228176418 shutdown_connections 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/4228176418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9c102250 0x7f6a9c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/4228176418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c101d10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/4228176418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c1078b0 0x7f6a9c107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 -- 172.21.15.64:0/4228176418 >> 172.21.15.64:0/4228176418 conn(0x7f6a9c0fd5f0 msgr2=0x7f6a9c0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 -- 172.21.15.64:0/4228176418 shutdown_connections 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 -- 172.21.15.64:0/4228176418 wait complete. 2024-08-19T01:30:45.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 Processor -- start 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 -- start start 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.177+0000 7f6aa36f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa36f2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c102250 0x7f6a9c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa36f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9c1078b0 0x7f6a9c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa36f2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c111cd0 con 0x7f6a9c1018b0 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa36f2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c111b50 con 0x7f6a9c102250 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa36f2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c111e50 con 0x7f6a9c1078b0 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34504/0 (socket says 172.21.15.64:34504) 2024-08-19T01:30:45.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 -- 172.21.15.64:0/1235853038 learned_addr learned my addr 172.21.15.64:0/1235853038 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa0c8d640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c102250 0x7f6a9c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa1c8f640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9c1078b0 0x7f6a9c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 -- 172.21.15.64:0/1235853038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9c1078b0 msgr2=0x7f6a9c19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9c1078b0 0x7f6a9c19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 -- 172.21.15.64:0/1235853038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c102250 msgr2=0x7f6a9c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c102250 0x7f6a9c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 -- 172.21.15.64:0/1235853038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c19d1c0 con 0x7f6a9c1018b0 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa1c8f640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9c1078b0 0x7f6a9c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa148e640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c198210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00eab0 tx=0x7f6a8c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6a9a7fc640 1 -- 172.21.15.64:0/1235853038 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c00ce40 con 0x7f6a9c1018b0 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9c19d4b0 con 0x7f6a9c1018b0 2024-08-19T01:30:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6a9a7fc640 1 -- 172.21.15.64:0/1235853038 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8c011950 con 0x7f6a9c1018b0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6a9a7fc640 1 -- 172.21.15.64:0/1235853038 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c010ae0 con 0x7f6a9c1018b0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.178+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a9c1a4d90 con 0x7f6a9c1018b0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.179+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a9c101d10 con 0x7f6a9c1018b0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.181+0000 7f6a9a7fc640 1 -- 172.21.15.64:0/1235853038 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a8c011b00 con 0x7f6a9c1018b0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.181+0000 7f6a9a7fc640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a840a2b80 0x7f6a840a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.181+0000 7f6aa0c8d640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a840a2b80 0x7f6a840a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.181+0000 7f6a9a7fc640 1 -- 172.21.15.64:0/1235853038 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f6a8c0cf260 con 0x7f6a9c1018b0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.182+0000 7f6aa0c8d640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a840a2b80 0x7f6a840a5040 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009900 tx=0x7f6a900058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.185+0000 7f6a9a7fc640 1 -- 172.21.15.64:0/1235853038 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6a8c016030 con 0x7f6a9c1018b0 2024-08-19T01:30:45.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.368+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6a9c109e40 con 0x7f6a9c1018b0 2024-08-19T01:30:45.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.369+0000 7f6a9a7fc640 1 -- 172.21.15.64:0/1235853038 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) ==== 84+0+65 (secure 0 0 0) 0x7f6a8c09c660 con 0x7f6a9c1018b0 2024-08-19T01:30:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:30:45.372 INFO:teuthology.orchestra.run.smithi064.stdout:[ 2024-08-19T01:30:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:30:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "poolnum": 1, 2024-08-19T01:30:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "poolname": ".mgr" 2024-08-19T01:30:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:30:45.372 INFO:teuthology.orchestra.run.smithi064.stdout:] 2024-08-19T01:30:45.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a840a2b80 msgr2=0x7f6a840a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:45.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a840a2b80 0x7f6a840a5040 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009900 tx=0x7f6a900058d0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 msgr2=0x7f6a9c198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c198210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00eab0 tx=0x7f6a8c00ef70 comp rx=0 tx=0).stop 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 shutdown_connections 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a840a2b80 0x7f6a840a5040 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9c1078b0 0x7f6a9c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c102250 0x7f6a9c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 --2- 172.21.15.64:0/1235853038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1018b0 0x7f6a9c198210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 >> 172.21.15.64:0/1235853038 conn(0x7f6a9c0fd5f0 msgr2=0x7f6a9c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 shutdown_connections 2024-08-19T01:30:45.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:45.372+0000 7f6aa36f2640 1 -- 172.21.15.64:0/1235853038 wait complete. 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:45 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1235853038' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-08-19T01:30:45.837 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:45.837 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.837 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.837 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:45.837 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:45.838 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.838 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:45.838 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:45 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1235853038' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-08-19T01:30:45.841 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:45.841 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.841 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.841 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:45.841 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:45.842 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:45.842 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:30:45.842 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:45 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1235853038' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-08-19T01:30:46.037 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-19T01:30:46.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-19T01:30:46.961 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:47.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 -- 172.21.15.64:0/3950470764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf4101160 msgr2=0x7f7cf410d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:47.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 --2- 172.21.15.64:0/3950470764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf4101160 0x7f7cf410d690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009cd0 tx=0x7f7cec02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:30:47.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 -- 172.21.15.64:0/3950470764 shutdown_connections 2024-08-19T01:30:47.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 --2- 172.21.15.64:0/3950470764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf4101160 0x7f7cf410d690 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 --2- 172.21.15.64:0/3950470764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cf41007c0 0x7f7cf4100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 --2- 172.21.15.64:0/3950470764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf41067c0 0x7f7cf4106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 -- 172.21.15.64:0/3950470764 >> 172.21.15.64:0/3950470764 conn(0x7f7cf40fc500 msgr2=0x7f7cf40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 -- 172.21.15.64:0/3950470764 shutdown_connections 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 -- 172.21.15.64:0/3950470764 wait complete. 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.450+0000 7f7cfa217640 1 Processor -- start 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cfa217640 1 -- start start 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cfa217640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cf41007c0 0x7f7cf4198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cfa217640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf4101160 0x7f7cf4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cfa217640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf41067c0 0x7f7cf419cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cfa217640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf4110c60 con 0x7f7cf41067c0 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cfa217640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf4110ae0 con 0x7f7cf4101160 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cfa217640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf4110de0 con 0x7f7cf41007c0 2024-08-19T01:30:47.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf41067c0 0x7f7cf419cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf41067c0 0x7f7cf419cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34526/0 (socket says 172.21.15.64:34526) 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 -- 172.21.15.64:0/2550057610 learned_addr learned my addr 172.21.15.64:0/2550057610 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf2ffd640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf4101160 0x7f7cf4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf37fe640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cf41007c0 0x7f7cf4198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 -- 172.21.15.64:0/2550057610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cf41007c0 msgr2=0x7f7cf4198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cf41007c0 0x7f7cf4198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 -- 172.21.15.64:0/2550057610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf4101160 msgr2=0x7f7cf4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf4101160 0x7f7cf4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 -- 172.21.15.64:0/2550057610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf419d2d0 con 0x7f7cf41067c0 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf37fe640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cf41007c0 0x7f7cf4198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.451+0000 7f7cf3fff640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf41067c0 0x7f7cf419cbf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7cec006010 tx=0x7f7cec003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.452+0000 7f7cf0ff9640 1 -- 172.21.15.64:0/2550057610 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec004450 con 0x7f7cf41067c0 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.452+0000 7f7cf0ff9640 1 -- 172.21.15.64:0/2550057610 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cec003e10 con 0x7f7cf41067c0 2024-08-19T01:30:47.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.452+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf419d560 con 0x7f7cf41067c0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.452+0000 7f7cf0ff9640 1 -- 172.21.15.64:0/2550057610 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec046480 con 0x7f7cf41067c0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.452+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf41a4ea0 con 0x7f7cf41067c0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.453+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf4100ca0 con 0x7f7cf41067c0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.454+0000 7f7cf0ff9640 1 -- 172.21.15.64:0/2550057610 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7cec0045f0 con 0x7f7cf41067c0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.454+0000 7f7cf0ff9640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7cdc0a2b80 0x7f7cdc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.455+0000 7f7cf37fe640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7cdc0a2b80 0x7f7cdc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.455+0000 7f7cf0ff9640 1 -- 172.21.15.64:0/2550057610 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f7cec02e030 con 0x7f7cf41067c0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.455+0000 7f7cf37fe640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7cdc0a2b80 0x7f7cdc0a5040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4006fd0 tx=0x7f7ce4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:47.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.459+0000 7f7cf0ff9640 1 -- 172.21.15.64:0/2550057610 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f7cec04b060 con 0x7f7cf41067c0 2024-08-19T01:30:47.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:47.650+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f7cf4109dc0 con 0x7f7cf41067c0 2024-08-19T01:30:47.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:47 smithi064 ceph-mon[18515]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:47.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:47 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:47 smithi123 ceph-mon[21829]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:47 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:47 smithi161 ceph-mon[21757]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:47.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:47.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:47 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.514+0000 7f7cf0ff9640 1 -- 172.21.15.64:0/2550057610 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) ==== 138+0+0 (secure 0 0 0) 0x7f7cec0bd950 con 0x7f7cf41067c0 2024-08-19T01:30:48.516 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_metadata' created 2024-08-19T01:30:48.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7cdc0a2b80 msgr2=0x7f7cdc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:48.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7cdc0a2b80 0x7f7cdc0a5040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4006fd0 tx=0x7f7ce4008040 comp rx=0 tx=0).stop 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf41067c0 msgr2=0x7f7cf419cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf41067c0 0x7f7cf419cbf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7cec006010 tx=0x7f7cec003c20 comp rx=0 tx=0).stop 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 shutdown_connections 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f7cdc0a2b80 0x7f7cdc0a5040 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cf41067c0 0x7f7cf419cbf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cf4101160 0x7f7cf4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 --2- 172.21.15.64:0/2550057610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cf41007c0 0x7f7cf4198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 >> 172.21.15.64:0/2550057610 conn(0x7f7cf40fc500 msgr2=0x7f7cf406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 shutdown_connections 2024-08-19T01:30:48.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:48.517+0000 7f7cfa217640 1 -- 172.21.15.64:0/2550057610 wait complete. 2024-08-19T01:30:48.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:48 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2550057610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-08-19T01:30:48.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:48.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:48.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:48 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2550057610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-08-19T01:30:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:48 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2550057610' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-08-19T01:30:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:30:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:30:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:30:49.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-08-19T01:30:49.551 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:49.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:49 smithi064 ceph-mon[18515]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:49.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:49 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2550057610' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-08-19T01:30:49.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:49 smithi064 ceph-mon[18515]: osdmap e79: 12 total, 12 up, 12 in 2024-08-19T01:30:49.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:49 smithi123 ceph-mon[21829]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:49 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2550057610' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-08-19T01:30:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:49 smithi123 ceph-mon[21829]: osdmap e79: 12 total, 12 up, 12 in 2024-08-19T01:30:49.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:49 smithi161 ceph-mon[21757]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:49.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:49 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2550057610' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-08-19T01:30:49.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:49 smithi161 ceph-mon[21757]: osdmap e79: 12 total, 12 up, 12 in 2024-08-19T01:30:50.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.054+0000 7f2869bed640 1 -- 172.21.15.64:0/3093855056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 msgr2=0x7f286410e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.054+0000 7f2869bed640 1 --2- 172.21.15.64:0/3093855056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f286410e780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f285c009cd0 tx=0x7f285c02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.054+0000 7f2869bed640 1 -- 172.21.15.64:0/3093855056 shutdown_connections 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.054+0000 7f2869bed640 1 --2- 172.21.15.64:0/3093855056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f286410e780 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.054+0000 7f2869bed640 1 --2- 172.21.15.64:0/3093855056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28641018b0 0x7f2864101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.054+0000 7f2869bed640 1 --2- 172.21.15.64:0/3093855056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28641078b0 0x7f2864107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.054+0000 7f2869bed640 1 -- 172.21.15.64:0/3093855056 >> 172.21.15.64:0/3093855056 conn(0x7f28640fd5f0 msgr2=0x7f28640ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.055+0000 7f2869bed640 1 -- 172.21.15.64:0/3093855056 shutdown_connections 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.055+0000 7f2869bed640 1 -- 172.21.15.64:0/3093855056 wait complete. 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.055+0000 7f2869bed640 1 Processor -- start 2024-08-19T01:30:50.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.055+0000 7f2869bed640 1 -- start start 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.055+0000 7f2869bed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28641018b0 0x7f2864074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.055+0000 7f2869bed640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f2864074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.055+0000 7f2869bed640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28641078b0 0x7f2864074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2869bed640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2864111d00 con 0x7f2864102250 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2869bed640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2864111b80 con 0x7f28641078b0 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2869bed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2864111e80 con 0x7f28641018b0 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f2864074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f2864074760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38088/0 (socket says 172.21.15.64:38088) 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 -- 172.21.15.64:0/170795261 learned_addr learned my addr 172.21.15.64:0/170795261 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:50.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2863fff640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28641078b0 0x7f2864074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f28637fe640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28641018b0 0x7f2864074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 -- 172.21.15.64:0/170795261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28641018b0 msgr2=0x7f2864074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28641018b0 0x7f2864074220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 -- 172.21.15.64:0/170795261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28641078b0 msgr2=0x7f2864074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28641078b0 0x7f2864074ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 -- 172.21.15.64:0/170795261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28641a6540 con 0x7f2864102250 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2862ffd640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f2864074760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f285800d950 tx=0x7f285800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2860ff9640 1 -- 172.21.15.64:0/170795261 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858017070 con 0x7f2864102250 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2860ff9640 1 -- 172.21.15.64:0/170795261 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f285800de80 con 0x7f2864102250 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28641a6830 con 0x7f2864102250 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2860ff9640 1 -- 172.21.15.64:0/170795261 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285800f620 con 0x7f2864102250 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.056+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28641a6d70 con 0x7f2864102250 2024-08-19T01:30:50.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.058+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2864101d10 con 0x7f2864102250 2024-08-19T01:30:50.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.058+0000 7f2860ff9640 1 -- 172.21.15.64:0/170795261 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2858009c30 con 0x7f2864102250 2024-08-19T01:30:50.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.059+0000 7f2860ff9640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f284c0a2b80 0x7f284c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:50.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.059+0000 7f28637fe640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f284c0a2b80 0x7f284c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:50.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.059+0000 7f2860ff9640 1 -- 172.21.15.64:0/170795261 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7497+0+0 (secure 0 0 0) 0x7f28580ce910 con 0x7f2864102250 2024-08-19T01:30:50.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.060+0000 7f28637fe640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f284c0a2b80 0x7f284c0a5040 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f285400ab60 tx=0x7f285400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:50.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.063+0000 7f2860ff9640 1 -- 172.21.15.64:0/170795261 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f285809bba0 con 0x7f2864102250 2024-08-19T01:30:50.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.238+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f28641a7090 con 0x7f2864102250 2024-08-19T01:30:50.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.529+0000 7f2860ff9640 1 -- 172.21.15.64:0/170795261 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v81) ==== 171+0+0 (secure 0 0 0) 0x7f285809b4a0 con 0x7f2864102250 2024-08-19T01:30:50.531 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data' created 2024-08-19T01:30:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f284c0a2b80 msgr2=0x7f284c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f284c0a2b80 0x7f284c0a5040 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f285400ab60 tx=0x7f285400a2d0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 msgr2=0x7f2864074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f2864074760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f285800d950 tx=0x7f285800dd50 comp rx=0 tx=0).stop 2024-08-19T01:30:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 shutdown_connections 2024-08-19T01:30:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f284c0a2b80 0x7f284c0a5040 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28641078b0 0x7f2864074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2864102250 0x7f2864074760 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 --2- 172.21.15.64:0/170795261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28641018b0 0x7f2864074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:50.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.532+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 >> 172.21.15.64:0/170795261 conn(0x7f28640fd5f0 msgr2=0x7f28640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:50.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.533+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 shutdown_connections 2024-08-19T01:30:50.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:50.533+0000 7f2869bed640 1 -- 172.21.15.64:0/170795261 wait complete. 2024-08-19T01:30:50.855 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:50 smithi064 ceph-mon[18515]: osdmap e80: 12 total, 12 up, 12 in 2024-08-19T01:30:50.855 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:50 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/170795261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-08-19T01:30:50.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:50 smithi123 ceph-mon[21829]: osdmap e80: 12 total, 12 up, 12 in 2024-08-19T01:30:50.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:50 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/170795261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-08-19T01:30:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:50 smithi161 ceph-mon[21757]: osdmap e80: 12 total, 12 up, 12 in 2024-08-19T01:30:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:50 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/170795261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-08-19T01:30:51.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-19T01:30:51.555 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:51.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:51 smithi064 ceph-mon[18515]: pgmap v258: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:51.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:51 smithi064 ceph-mon[18515]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T01:30:51.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:51 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/170795261' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-08-19T01:30:51.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:51 smithi064 ceph-mon[18515]: osdmap e81: 12 total, 12 up, 12 in 2024-08-19T01:30:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:51 smithi123 ceph-mon[21829]: pgmap v258: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:51 smithi123 ceph-mon[21829]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T01:30:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:51 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/170795261' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-08-19T01:30:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:51 smithi123 ceph-mon[21829]: osdmap e81: 12 total, 12 up, 12 in 2024-08-19T01:30:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:51 smithi161 ceph-mon[21757]: pgmap v258: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:51 smithi161 ceph-mon[21757]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T01:30:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:51 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/170795261' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-08-19T01:30:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:51 smithi161 ceph-mon[21757]: osdmap e81: 12 total, 12 up, 12 in 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 -- 172.21.15.64:0/1669205254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc1033f0 msgr2=0x7f5adc10b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/1669205254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc1033f0 0x7f5adc10b8e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009900 tx=0x7f5ac402a480 comp rx=0 tx=0).stop 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 -- 172.21.15.64:0/1669205254 shutdown_connections 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/1669205254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc10beb0 0x7f5adc10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/1669205254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc1033f0 0x7f5adc10b8e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/1669205254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc102ad0 0x7f5adc102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 -- 172.21.15.64:0/1669205254 >> 172.21.15.64:0/1669205254 conn(0x7f5adc0fc4d0 msgr2=0x7f5adc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 -- 172.21.15.64:0/1669205254 shutdown_connections 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.058+0000 7f5ae09aa640 1 -- 172.21.15.64:0/1669205254 wait complete. 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 Processor -- start 2024-08-19T01:30:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 -- start start 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111d50 con 0x7f5adc10beb0 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111bd0 con 0x7f5adc1033f0 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5ae09aa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111ed0 con 0x7f5adc102ad0 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.059+0000 7f5adad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc19cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38114/0 (socket says 172.21.15.64:38114) 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 -- 172.21.15.64:0/911032235 learned_addr learned my addr 172.21.15.64:0/911032235 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ada59c640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ad1d9b640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 -- 172.21.15.64:0/911032235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 msgr2=0x7f5adc198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:52.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 -- 172.21.15.64:0/911032235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 msgr2=0x7f5adc198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 -- 172.21.15.64:0/911032235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc19d2e0 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ada59c640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5adad9d640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc19cc00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b970 tx=0x7f5ad400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ad3fff640 1 -- 172.21.15.64:0/911032235 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4019070 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ad3fff640 1 -- 172.21.15.64:0/911032235 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ad400cb40 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc19d5d0 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ad3fff640 1 -- 172.21.15.64:0/911032235 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4007650 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.060+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc1a4eb0 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.061+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5adc102f30 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.063+0000 7f5ad3fff640 1 -- 172.21.15.64:0/911032235 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5ad401c070 con 0x7f5adc10beb0 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.063+0000 7f5ad3fff640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5ac80a2b80 0x7f5ac80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:52.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.063+0000 7f5ada59c640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5ac80a2b80 0x7f5ac80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:52.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.063+0000 7f5ad3fff640 1 -- 172.21.15.64:0/911032235 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7879+0+0 (secure 0 0 0) 0x7f5ad40ce9a0 con 0x7f5adc10beb0 2024-08-19T01:30:52.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.064+0000 7f5ada59c640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5ac80a2b80 0x7f5ac80a5040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5acc006fd0 tx=0x7f5acc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:52.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.067+0000 7f5ad3fff640 1 -- 172.21.15.64:0/911032235 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f5ad409bab0 con 0x7f5adc10beb0 2024-08-19T01:30:52.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.280+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5adc199690 con 0x7f5adc10beb0 2024-08-19T01:30:52.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.544+0000 7f5ad3fff640 1 -- 172.21.15.64:0/911032235 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f5ad409b3b0 con 0x7f5adc10beb0 2024-08-19T01:30:52.546 INFO:teuthology.orchestra.run.smithi064.stderr:new fs with metadata pool 2 and data pool 3 2024-08-19T01:30:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5ac80a2b80 msgr2=0x7f5ac80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5ac80a2b80 0x7f5ac80a5040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5acc006fd0 tx=0x7f5acc008040 comp rx=0 tx=0).stop 2024-08-19T01:30:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 msgr2=0x7f5adc19cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc19cc00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b970 tx=0x7f5ad400be30 comp rx=0 tx=0).stop 2024-08-19T01:30:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 shutdown_connections 2024-08-19T01:30:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f5ac80a2b80 0x7f5ac80a5040 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc19cc00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.546+0000 7f5ae09aa640 1 --2- 172.21.15.64:0/911032235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.547+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 >> 172.21.15.64:0/911032235 conn(0x7f5adc0fc4d0 msgr2=0x7f5adc0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.547+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 shutdown_connections 2024-08-19T01:30:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:52.547+0000 7f5ae09aa640 1 -- 172.21.15.64:0/911032235 wait complete. 2024-08-19T01:30:52.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:52 smithi064 ceph-mon[18515]: osdmap e82: 12 total, 12 up, 12 in 2024-08-19T01:30:52.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:52 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/911032235' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-08-19T01:30:52.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:52 smithi064 ceph-mon[18515]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-08-19T01:30:52.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:52 smithi123 ceph-mon[21829]: osdmap e82: 12 total, 12 up, 12 in 2024-08-19T01:30:52.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:52 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/911032235' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-08-19T01:30:52.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:52 smithi123 ceph-mon[21829]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-08-19T01:30:52.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:52 smithi161 ceph-mon[21757]: osdmap e82: 12 total, 12 up, 12 in 2024-08-19T01:30:52.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:52 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/911032235' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-08-19T01:30:52.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:52 smithi161 ceph-mon[21757]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-08-19T01:30:53.330 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-08-19T01:30:53.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-08-19T01:30:53.589 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:53.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:53 smithi064 ceph-mon[18515]: pgmap v261: 97 pgs: 14 creating+peering, 66 unknown, 17 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:53.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:53 smithi064 ceph-mon[18515]: osdmap e83: 12 total, 12 up, 12 in 2024-08-19T01:30:53.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:53 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/911032235' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-08-19T01:30:53.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:53 smithi064 ceph-mon[18515]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-08-19T01:30:53.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:53 smithi064 ceph-mon[18515]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-08-19T01:30:53.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:53 smithi123 ceph-mon[21829]: pgmap v261: 97 pgs: 14 creating+peering, 66 unknown, 17 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:53 smithi123 ceph-mon[21829]: osdmap e83: 12 total, 12 up, 12 in 2024-08-19T01:30:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:53 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/911032235' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-08-19T01:30:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:53 smithi123 ceph-mon[21829]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-08-19T01:30:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:53 smithi123 ceph-mon[21829]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-08-19T01:30:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:53 smithi161 ceph-mon[21757]: pgmap v261: 97 pgs: 14 creating+peering, 66 unknown, 17 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:30:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:53 smithi161 ceph-mon[21757]: osdmap e83: 12 total, 12 up, 12 in 2024-08-19T01:30:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:53 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/911032235' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-08-19T01:30:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:53 smithi161 ceph-mon[21757]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-08-19T01:30:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:53 smithi161 ceph-mon[21757]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-08-19T01:30:54.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 -- 172.21.15.64:0/919146671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8581067b0 msgr2=0x7ff858106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 --2- 172.21.15.64:0/919146671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8581067b0 0x7ff858106b90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff848009930 tx=0x7ff84802a420 comp rx=0 tx=0).stop 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 -- 172.21.15.64:0/919146671 shutdown_connections 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 --2- 172.21.15.64:0/919146671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff858101150 0x7ff85810d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 --2- 172.21.15.64:0/919146671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8581007b0 0x7ff858100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 --2- 172.21.15.64:0/919146671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8581067b0 0x7ff858106b90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 -- 172.21.15.64:0/919146671 >> 172.21.15.64:0/919146671 conn(0x7ff8580fc4d0 msgr2=0x7ff8580fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 -- 172.21.15.64:0/919146671 shutdown_connections 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.104+0000 7ff85cf35640 1 -- 172.21.15.64:0/919146671 wait complete. 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 Processor -- start 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 -- start start 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8581007b0 0x7ff858198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff858101150 0x7ff858198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:54.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8581067b0 0x7ff85819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff858111cd0 con 0x7ff858101150 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff858111b50 con 0x7ff8581007b0 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85cf35640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff858111e50 con 0x7ff8581067b0 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff855d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff858101150 0x7ff858198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff855d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff858101150 0x7ff858198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38132/0 (socket says 172.21.15.64:38132) 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff855d9b640 1 -- 172.21.15.64:0/1519174164 learned_addr learned my addr 172.21.15.64:0/1519174164 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff85659c640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8581007b0 0x7ff858198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff855d9b640 1 -- 172.21.15.64:0/1519174164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8581067b0 msgr2=0x7ff85819cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff855d9b640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8581067b0 0x7ff85819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff855d9b640 1 -- 172.21.15.64:0/1519174164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8581007b0 msgr2=0x7ff858198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.105+0000 7ff855d9b640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8581007b0 0x7ff858198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff855d9b640 1 -- 172.21.15.64:0/1519174164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85819d1e0 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff85659c640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8581007b0 0x7ff858198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff855d9b640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff858101150 0x7ff858198770 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00b4a0 tx=0x7ff84c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff8477fe640 1 -- 172.21.15.64:0/1519174164 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c014070 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff8477fe640 1 -- 172.21.15.64:0/1519174164 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84c010af0 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff8477fe640 1 -- 172.21.15.64:0/1519174164 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c017d10 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85819d4d0 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.106+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8581a4db0 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.107+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff858100c10 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.108+0000 7ff8477fe640 1 -- 172.21.15.64:0/1519174164 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff84c017070 con 0x7ff858101150 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.109+0000 7ff8477fe640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8400a2c00 0x7ff8400a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.109+0000 7ff85659c640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8400a2c00 0x7ff8400a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.109+0000 7ff8477fe640 1 -- 172.21.15.64:0/1519174164 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7979+0+0 (secure 0 0 0) 0x7ff84c0ce700 con 0x7ff858101150 2024-08-19T01:30:54.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.109+0000 7ff85659c640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8400a2c00 0x7ff8400a50c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff848000c00 tx=0x7ff848032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:54.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.112+0000 7ff8477fe640 1 -- 172.21.15.64:0/1519174164 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff84c09b7b0 con 0x7ff858101150 2024-08-19T01:30:54.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.295+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7ff858199590 con 0x7ff858101150 2024-08-19T01:30:54.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.559+0000 7ff8477fe640 1 -- 172.21.15.64:0/1519174164 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v84) ==== 157+0+0 (secure 0 0 0) 0x7ff858199590 con 0x7ff858101150 2024-08-19T01:30:54.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8400a2c00 msgr2=0x7ff8400a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:54.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8400a2c00 0x7ff8400a50c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff848000c00 tx=0x7ff848032d50 comp rx=0 tx=0).stop 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff858101150 msgr2=0x7ff858198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff858101150 0x7ff858198770 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00b4a0 tx=0x7ff84c00b960 comp rx=0 tx=0).stop 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 shutdown_connections 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8400a2c00 0x7ff8400a50c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8581067b0 0x7ff85819cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff858101150 0x7ff858198770 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 --2- 172.21.15.64:0/1519174164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8581007b0 0x7ff858198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 >> 172.21.15.64:0/1519174164 conn(0x7ff8580fc4d0 msgr2=0x7ff8580fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 shutdown_connections 2024-08-19T01:30:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:54.562+0000 7ff85cf35640 1 -- 172.21.15.64:0/1519174164 wait complete. 2024-08-19T01:30:54.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:54 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] up:active 2024-08-19T01:30:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:54 smithi123 ceph-mon[21829]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:30:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:54 smithi123 ceph-mon[21829]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:30:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:54 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1519174164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-08-19T01:30:54.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:54 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] up:active 2024-08-19T01:30:54.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:54 smithi064 ceph-mon[18515]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:30:54.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:54 smithi064 ceph-mon[18515]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:30:54.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:54 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1519174164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-08-19T01:30:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:54 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] up:active 2024-08-19T01:30:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:54 smithi161 ceph-mon[21757]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:30:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:54 smithi161 ceph-mon[21757]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:30:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:54 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1519174164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-08-19T01:30:55.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-08-19T01:30:55.506 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:55.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:55 smithi123 ceph-mon[21829]: pgmap v263: 97 pgs: 14 creating+peering, 14 unknown, 69 active+clean; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-08-19T01:30:55.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:55 smithi123 ceph-mon[21829]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-08-19T01:30:55.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:55 smithi123 ceph-mon[21829]: Cluster is now healthy 2024-08-19T01:30:55.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:55 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1519174164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-08-19T01:30:55.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:55 smithi123 ceph-mon[21829]: osdmap e84: 12 total, 12 up, 12 in 2024-08-19T01:30:55.850 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:55 smithi161 ceph-mon[21757]: pgmap v263: 97 pgs: 14 creating+peering, 14 unknown, 69 active+clean; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-08-19T01:30:55.851 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:55 smithi161 ceph-mon[21757]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-08-19T01:30:55.851 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:55 smithi161 ceph-mon[21757]: Cluster is now healthy 2024-08-19T01:30:55.851 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:55 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1519174164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-08-19T01:30:55.851 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:55 smithi161 ceph-mon[21757]: osdmap e84: 12 total, 12 up, 12 in 2024-08-19T01:30:55.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:55 smithi064 ceph-mon[18515]: pgmap v263: 97 pgs: 14 creating+peering, 14 unknown, 69 active+clean; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-08-19T01:30:55.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:55 smithi064 ceph-mon[18515]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-08-19T01:30:55.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:55 smithi064 ceph-mon[18515]: Cluster is now healthy 2024-08-19T01:30:55.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:55 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1519174164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-08-19T01:30:55.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:55 smithi064 ceph-mon[18515]: osdmap e84: 12 total, 12 up, 12 in 2024-08-19T01:30:56.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 -- 172.21.15.64:0/2096488774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e81007b0 msgr2=0x7f47e8100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:56.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 --2- 172.21.15.64:0/2096488774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e81007b0 0x7f47e8100c10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009930 tx=0x7f47d002a420 comp rx=0 tx=0).stop 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 -- 172.21.15.64:0/2096488774 shutdown_connections 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 --2- 172.21.15.64:0/2096488774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e8101150 0x7f47e810d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 --2- 172.21.15.64:0/2096488774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e81007b0 0x7f47e8100c10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 --2- 172.21.15.64:0/2096488774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47e81067b0 0x7f47e8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 -- 172.21.15.64:0/2096488774 >> 172.21.15.64:0/2096488774 conn(0x7f47e80fc4d0 msgr2=0x7f47e80fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 -- 172.21.15.64:0/2096488774 shutdown_connections 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 -- 172.21.15.64:0/2096488774 wait complete. 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.016+0000 7f47edd76640 1 Processor -- start 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47edd76640 1 -- start start 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47edd76640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47e81007b0 0x7f47e8198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47edd76640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e8101150 0x7f47e81994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:56.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47edd76640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e81067b0 0x7f47e81a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47edd76640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f47e8110bb0 con 0x7f47e8101150 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47edd76640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47e8110a30 con 0x7f47e81007b0 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47edd76640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47e8110d30 con 0x7f47e81067b0 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e8101150 0x7f47e81994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e8101150 0x7f47e81994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38152/0 (socket says 172.21.15.64:38152) 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 -- 172.21.15.64:0/4146499426 learned_addr learned my addr 172.21.15.64:0/4146499426 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e77fe640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47e81007b0 0x7f47e8198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e7fff640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e81067b0 0x7f47e81a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 -- 172.21.15.64:0/4146499426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e81067b0 msgr2=0x7f47e81a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e81067b0 0x7f47e81a3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 -- 172.21.15.64:0/4146499426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47e81007b0 msgr2=0x7f47e8198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:56.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47e81007b0 0x7f47e8198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 -- 172.21.15.64:0/4146499426 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e81a4520 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.017+0000 7f47e6ffd640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e8101150 0x7f47e81994a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009230 tx=0x7f47d0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.018+0000 7f47e4ff9640 1 -- 172.21.15.64:0/4146499426 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d003f030 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.018+0000 7f47e4ff9640 1 -- 172.21.15.64:0/4146499426 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d0003ad0 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.018+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e81a46f0 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.018+0000 7f47e4ff9640 1 -- 172.21.15.64:0/4146499426 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d0035070 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.018+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e81a4b40 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.019+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e8100c10 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.020+0000 7f47e4ff9640 1 -- 172.21.15.64:0/4146499426 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f47d0042030 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.020+0000 7f47e4ff9640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47d40a2b80 0x7f47d40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.021+0000 7f47e77fe640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47d40a2b80 0x7f47d40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.021+0000 7f47e4ff9640 1 -- 172.21.15.64:0/4146499426 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8277+0+0 (secure 0 0 0) 0x7f47d00ef950 con 0x7f47e8101150 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.021+0000 7f47e77fe640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47d40a2b80 0x7f47d40a5040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f47d800ab60 tx=0x7f47d800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:56.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.024+0000 7f47e4ff9640 1 -- 172.21.15.64:0/4146499426 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f47d00bc880 con 0x7f47e8101150 2024-08-19T01:30:56.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:56.200+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f47e81a4e60 con 0x7f47e8101150 2024-08-19T01:30:56.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:56 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4146499426' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-08-19T01:30:56.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:56 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4146499426' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-08-19T01:30:56.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:56 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4146499426' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-08-19T01:30:57.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.565+0000 7f47e4ff9640 1 -- 172.21.15.64:0/4146499426 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v86) ==== 228+0+0 (secure 0 0 0) 0x7f47d00bc180 con 0x7f47e8101150 2024-08-19T01:30:57.566 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data_ec' created 2024-08-19T01:30:57.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47d40a2b80 msgr2=0x7f47d40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:57.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47d40a2b80 0x7f47d40a5040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f47d800ab60 tx=0x7f47d800a2d0 comp rx=0 tx=0).stop 2024-08-19T01:30:57.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e8101150 msgr2=0x7f47e81994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:57.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e8101150 0x7f47e81994a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009230 tx=0x7f47d0004270 comp rx=0 tx=0).stop 2024-08-19T01:30:57.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 shutdown_connections 2024-08-19T01:30:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47d40a2b80 0x7f47d40a5040 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e81067b0 0x7f47e81a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47e8101150 0x7f47e81994a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 --2- 172.21.15.64:0/4146499426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47e81007b0 0x7f47e8198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.567+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 >> 172.21.15.64:0/4146499426 conn(0x7f47e80fc4d0 msgr2=0x7f47e80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.568+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 shutdown_connections 2024-08-19T01:30:57.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:57.568+0000 7f47edd76640 1 -- 172.21.15.64:0/4146499426 wait complete. 2024-08-19T01:30:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:57 smithi123 ceph-mon[21829]: pgmap v265: 97 pgs: 4 creating+peering, 93 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-08-19T01:30:57.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:57 smithi123 ceph-mon[21829]: osdmap e85: 12 total, 12 up, 12 in 2024-08-19T01:30:57.666 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:30:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: 2024-08-19T01:30:57.344+0000 7ff772445640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-19T01:30:57.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:57 smithi064 ceph-mon[18515]: pgmap v265: 97 pgs: 4 creating+peering, 93 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-08-19T01:30:57.882 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:57 smithi064 ceph-mon[18515]: osdmap e85: 12 total, 12 up, 12 in 2024-08-19T01:30:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:57 smithi161 ceph-mon[21757]: pgmap v265: 97 pgs: 4 creating+peering, 93 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-08-19T01:30:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:57 smithi161 ceph-mon[21757]: osdmap e85: 12 total, 12 up, 12 in 2024-08-19T01:30:58.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-08-19T01:30:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:30:58.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:58 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4146499426' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-08-19T01:30:58.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:58 smithi064 ceph-mon[18515]: osdmap e86: 12 total, 12 up, 12 in 2024-08-19T01:30:58.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:58 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4146499426' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-08-19T01:30:58.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:58 smithi123 ceph-mon[21829]: osdmap e86: 12 total, 12 up, 12 in 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 -- 172.21.15.64:0/2848000273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 msgr2=0x7fed3010b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 --2- 172.21.15.64:0/2848000273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed3010b8e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fed18009900 tx=0x7fed1802a480 comp rx=0 tx=0).stop 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 -- 172.21.15.64:0/2848000273 shutdown_connections 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 --2- 172.21.15.64:0/2848000273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed3010beb0 0x7fed3010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 --2- 172.21.15.64:0/2848000273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed3010b8e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 --2- 172.21.15.64:0/2848000273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed30102ad0 0x7fed30102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 -- 172.21.15.64:0/2848000273 >> 172.21.15.64:0/2848000273 conn(0x7fed300fc4d0 msgr2=0x7fed300fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 -- 172.21.15.64:0/2848000273 shutdown_connections 2024-08-19T01:30:58.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.927+0000 7fed36909640 1 -- 172.21.15.64:0/2848000273 wait complete. 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 Processor -- start 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 -- start start 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed30102ad0 0x7fed301981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed301986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed3010beb0 0x7fed3019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fed30111c90 con 0x7fed301033f0 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fed30111b10 con 0x7fed3010beb0 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed36909640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed30111e10 con 0x7fed30102ad0 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed27fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed301986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed27fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed301986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47962/0 (socket says 172.21.15.64:47962) 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.928+0000 7fed27fff640 1 -- 172.21.15.64:0/3393129498 learned_addr learned my addr 172.21.15.64:0/3393129498 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:30:58.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed2ffff640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed30102ad0 0x7fed301981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed34ea6640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed3010beb0 0x7fed3019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed27fff640 1 -- 172.21.15.64:0/3393129498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed30102ad0 msgr2=0x7fed301981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed27fff640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed30102ad0 0x7fed301981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed27fff640 1 -- 172.21.15.64:0/3393129498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed3010beb0 msgr2=0x7fed3019ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed27fff640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed3010beb0 0x7fed3019ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed27fff640 1 -- 172.21.15.64:0/3393129498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3019d160 con 0x7fed301033f0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed34ea6640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed3010beb0 0x7fed3019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed27fff640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed301986f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fed18006010 tx=0x7fed180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed2dffb640 1 -- 172.21.15.64:0/3393129498 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1803f030 con 0x7fed301033f0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3019d3f0 con 0x7fed301033f0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed2dffb640 1 -- 172.21.15.64:0/3393129498 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed18003ce0 con 0x7fed301033f0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed2dffb640 1 -- 172.21.15.64:0/3393129498 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18035070 con 0x7fed301033f0 2024-08-19T01:30:58.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.929+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed301a4d30 con 0x7fed301033f0 2024-08-19T01:30:58.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.930+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3006ad40 con 0x7fed301033f0 2024-08-19T01:30:58.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.932+0000 7fed2dffb640 1 -- 172.21.15.64:0/3393129498 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fed18042030 con 0x7fed301033f0 2024-08-19T01:30:58.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.932+0000 7fed2dffb640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fed1c0a2b80 0x7fed1c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:30:58.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.933+0000 7fed2ffff640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fed1c0a2b80 0x7fed1c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:30:58.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.933+0000 7fed2dffb640 1 -- 172.21.15.64:0/3393129498 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fed180efc80 con 0x7fed301033f0 2024-08-19T01:30:58.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.933+0000 7fed2ffff640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fed1c0a2b80 0x7fed1c0a5040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed20005fd0 tx=0x7fed20005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:30:58.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:58.936+0000 7fed2dffb640 1 -- 172.21.15.64:0/3393129498 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fed180bca80 con 0x7fed301033f0 2024-08-19T01:30:58.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:58 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4146499426' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-08-19T01:30:58.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:58 smithi161 ceph-mon[21757]: osdmap e86: 12 total, 12 up, 12 in 2024-08-19T01:30:59.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.110+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fed30106ef0 con 0x7fed301033f0 2024-08-19T01:30:59.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.577+0000 7fed2dffb640 1 -- 172.21.15.64:0/3393129498 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fed180bc380 con 0x7fed301033f0 2024-08-19T01:30:59.578 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 allow_ec_overwrites to true 2024-08-19T01:30:59.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.579+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fed1c0a2b80 msgr2=0x7fed1c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:59.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.579+0000 7fed36909640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fed1c0a2b80 0x7fed1c0a5040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed20005fd0 tx=0x7fed20005850 comp rx=0 tx=0).stop 2024-08-19T01:30:59.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.579+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 msgr2=0x7fed301986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:30:59.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.579+0000 7fed36909640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed301986f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fed18006010 tx=0x7fed180042f0 comp rx=0 tx=0).stop 2024-08-19T01:30:59.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 shutdown_connections 2024-08-19T01:30:59.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fed1c0a2b80 0x7fed1c0a5040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:59.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed3010beb0 0x7fed3019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:59.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed301033f0 0x7fed301986f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:59.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 --2- 172.21.15.64:0/3393129498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed30102ad0 0x7fed301981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:30:59.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 >> 172.21.15.64:0/3393129498 conn(0x7fed300fc4d0 msgr2=0x7fed300fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:30:59.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 shutdown_connections 2024-08-19T01:30:59.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:30:59.580+0000 7fed36909640 1 -- 172.21.15.64:0/3393129498 wait complete. 2024-08-19T01:30:59.907 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:59 smithi064 ceph-mon[18515]: pgmap v268: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-08-19T01:30:59.907 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:59 smithi064 ceph-mon[18515]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T01:30:59.907 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:59 smithi064 ceph-mon[18515]: osdmap e87: 12 total, 12 up, 12 in 2024-08-19T01:30:59.907 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:30:59 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3393129498' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-08-19T01:30:59.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:59 smithi123 ceph-mon[21829]: pgmap v268: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-08-19T01:30:59.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:59 smithi123 ceph-mon[21829]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T01:30:59.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:59 smithi123 ceph-mon[21829]: osdmap e87: 12 total, 12 up, 12 in 2024-08-19T01:30:59.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:30:59 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3393129498' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-08-19T01:30:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:59 smithi161 ceph-mon[21757]: pgmap v268: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-08-19T01:30:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:59 smithi161 ceph-mon[21757]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T01:30:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:59 smithi161 ceph-mon[21757]: osdmap e87: 12 total, 12 up, 12 in 2024-08-19T01:30:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:30:59 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3393129498' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-08-19T01:31:00.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-08-19T01:31:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:00.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:00 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3393129498' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-08-19T01:31:00.762 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:00 smithi064 ceph-mon[18515]: osdmap e88: 12 total, 12 up, 12 in 2024-08-19T01:31:00.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:00 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3393129498' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-08-19T01:31:00.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:00 smithi123 ceph-mon[21829]: osdmap e88: 12 total, 12 up, 12 in 2024-08-19T01:31:00.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:00 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3393129498' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-08-19T01:31:00.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:00 smithi161 ceph-mon[21757]: osdmap e88: 12 total, 12 up, 12 in 2024-08-19T01:31:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 -- 172.21.15.64:0/354739749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 msgr2=0x7f3144100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 --2- 172.21.15.64:0/354739749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144100290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3134009900 tx=0x7f313402a410 comp rx=0 tx=0).stop 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 -- 172.21.15.64:0/354739749 shutdown_connections 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 --2- 172.21.15.64:0/354739749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31441007d0 0x7f314410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 --2- 172.21.15.64:0/354739749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144100290 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 --2- 172.21.15.64:0/354739749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31441068b0 0x7f3144106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 -- 172.21.15.64:0/354739749 >> 172.21.15.64:0/354739749 conn(0x7f3144076f60 msgr2=0x7f31440fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 -- 172.21.15.64:0/354739749 shutdown_connections 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.993+0000 7f314c512640 1 -- 172.21.15.64:0/354739749 wait complete. 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 Processor -- start 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 -- start start 2024-08-19T01:31:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31441007d0 0x7f31441986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31441068b0 0x7f314419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111d80 con 0x7f31440ffe30 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111c00 con 0x7f31441068b0 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314c512640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111f00 con 0x7f31441007d0 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314a2ae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314a2ae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47978/0 (socket says 172.21.15.64:47978) 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.994+0000 7f314a2ae640 1 -- 172.21.15.64:0/2205190134 learned_addr learned my addr 172.21.15.64:0/2205190134 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f3149aad640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31441007d0 0x7f31441986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314a2ae640 1 -- 172.21.15.64:0/2205190134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31441007d0 msgr2=0x7f31441986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314a2ae640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31441007d0 0x7f31441986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314a2ae640 1 -- 172.21.15.64:0/2205190134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31441068b0 msgr2=0x7f314419ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314a2ae640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31441068b0 0x7f314419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314a2ae640 1 -- 172.21.15.64:0/2205190134 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314419d110 con 0x7f31440ffe30 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314a2ae640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144198160 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f314000c920 tx=0x7f314000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f313b7fe640 1 -- 172.21.15.64:0/2205190134 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3140007c70 con 0x7f31440ffe30 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f313b7fe640 1 -- 172.21.15.64:0/2205190134 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3140007e10 con 0x7f31440ffe30 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314419d400 con 0x7f31440ffe30 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f313b7fe640 1 -- 172.21.15.64:0/2205190134 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3140018890 con 0x7f31440ffe30 2024-08-19T01:31:00.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.995+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31441a4ce0 con 0x7f31440ffe30 2024-08-19T01:31:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.996+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3144100290 con 0x7f31440ffe30 2024-08-19T01:31:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.997+0000 7f313b7fe640 1 -- 172.21.15.64:0/2205190134 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3140018a70 con 0x7f31440ffe30 2024-08-19T01:31:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.998+0000 7f313b7fe640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f31300a2b80 0x7f31300a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.998+0000 7f3149aad640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f31300a2b80 0x7f31300a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.998+0000 7f313b7fe640 1 -- 172.21.15.64:0/2205190134 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f31400cf4c0 con 0x7f31440ffe30 2024-08-19T01:31:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:00.998+0000 7f3149aad640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f31300a2b80 0x7f31300a5040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f31440ff440 tx=0x7f3134005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.002+0000 7f313b7fe640 1 -- 172.21.15.64:0/2205190134 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f3140014030 con 0x7f31440ffe30 2024-08-19T01:31:01.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.214+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f31441a4f10 con 0x7f31440ffe30 2024-08-19T01:31:01.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.583+0000 7f313b7fe640 1 -- 172.21.15.64:0/2205190134 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f314000de20 con 0x7f31440ffe30 2024-08-19T01:31:01.585 INFO:teuthology.orchestra.run.smithi064.stderr:added data pool 4 to fsmap 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f31300a2b80 msgr2=0x7f31300a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f31300a2b80 0x7f31300a5040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f31440ff440 tx=0x7f3134005a20 comp rx=0 tx=0).stop 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 msgr2=0x7f3144198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144198160 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f314000c920 tx=0x7f314000cde0 comp rx=0 tx=0).stop 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 shutdown_connections 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f31300a2b80 0x7f31300a5040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31441068b0 0x7f314419ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31441007d0 0x7f31441986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 --2- 172.21.15.64:0/2205190134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31440ffe30 0x7f3144198160 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 >> 172.21.15.64:0/2205190134 conn(0x7f3144076f60 msgr2=0x7f31441047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:01.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 shutdown_connections 2024-08-19T01:31:01.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:01.586+0000 7f314c512640 1 -- 172.21.15.64:0/2205190134 wait complete. 2024-08-19T01:31:01.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:01 smithi064 ceph-mon[18515]: pgmap v271: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-08-19T01:31:01.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:01 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2205190134' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-08-19T01:31:01.920 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:01 smithi123 ceph-mon[21829]: pgmap v271: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-08-19T01:31:01.920 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:01 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2205190134' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-08-19T01:31:01.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:01 smithi161 ceph-mon[21757]: pgmap v271: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-08-19T01:31:01.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:01 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2205190134' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-08-19T01:31:02.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:31:02.509 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:02.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:02 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2205190134' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-08-19T01:31:02.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:02 smithi064 ceph-mon[18515]: osdmap e89: 12 total, 12 up, 12 in 2024-08-19T01:31:02.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:02 smithi064 ceph-mon[18515]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:02.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:02 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2205190134' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-08-19T01:31:02.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:02 smithi123 ceph-mon[21829]: osdmap e89: 12 total, 12 up, 12 in 2024-08-19T01:31:02.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:02 smithi123 ceph-mon[21829]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:02 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2205190134' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-08-19T01:31:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:02 smithi161 ceph-mon[21757]: osdmap e89: 12 total, 12 up, 12 in 2024-08-19T01:31:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:02 smithi161 ceph-mon[21757]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:03.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 -- 172.21.15.64:0/3346856350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec01009f0 msgr2=0x7f2ec0106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:03.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/3346856350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec01009f0 0x7f2ec0106530 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009900 tx=0x7f2eb402a410 comp rx=0 tx=0).stop 2024-08-19T01:31:03.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 -- 172.21.15.64:0/3346856350 shutdown_connections 2024-08-19T01:31:03.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/3346856350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ec0106c60 0x7f2ec010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/3346856350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec01009f0 0x7f2ec0106530 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/3346856350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ec0100040 0x7f2ec0100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 -- 172.21.15.64:0/3346856350 >> 172.21.15.64:0/3346856350 conn(0x7f2ec0076f40 msgr2=0x7f2ec00fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 -- 172.21.15.64:0/3346856350 shutdown_connections 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.985+0000 7f2ec5b98640 1 -- 172.21.15.64:0/3346856350 wait complete. 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 Processor -- start 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 -- start start 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ec0100040 0x7f2ec019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ec01009f0 0x7f2ec019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec0106c60 0x7f2ec019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0111d60 con 0x7f2ec0106c60 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0111be0 con 0x7f2ec01009f0 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ec5b98640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0111ee0 con 0x7f2ec0100040 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ebffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec0106c60 0x7f2ec019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:03.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ebffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec0106c60 0x7f2ec019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47994/0 (socket says 172.21.15.64:47994) 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ebffff640 1 -- 172.21.15.64:0/2943307487 learned_addr learned my addr 172.21.15.64:0/2943307487 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ebffff640 1 -- 172.21.15.64:0/2943307487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ec0100040 msgr2=0x7f2ec019a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.986+0000 7f2ebffff640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ec0100040 0x7f2ec019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebffff640 1 -- 172.21.15.64:0/2943307487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ec01009f0 msgr2=0x7f2ec019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebeffd640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ec01009f0 0x7f2ec019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebffff640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ec01009f0 0x7f2ec019aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebffff640 1 -- 172.21.15.64:0/2943307487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec019f4b0 con 0x7f2ec0106c60 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebffff640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec0106c60 0x7f2ec019edd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800b970 tx=0x7f2eb800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebcff9640 1 -- 172.21.15.64:0/2943307487 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8019070 con 0x7f2ec0106c60 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec019f7a0 con 0x7f2ec0106c60 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebcff9640 1 -- 172.21.15.64:0/2943307487 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb800cb40 con 0x7f2ec0106c60 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ebcff9640 1 -- 172.21.15.64:0/2943307487 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8007610 con 0x7f2ec0106c60 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.987+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec01a7080 con 0x7f2ec0106c60 2024-08-19T01:31:03.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.988+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec01004a0 con 0x7f2ec0106c60 2024-08-19T01:31:03.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.989+0000 7f2ebcff9640 1 -- 172.21.15.64:0/2943307487 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2eb801c070 con 0x7f2ec0106c60 2024-08-19T01:31:03.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.990+0000 7f2ebcff9640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ea80a2b80 0x7f2ea80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:03.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.990+0000 7f2ebf7fe640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ea80a2b80 0x7f2ea80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:03.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.990+0000 7f2ebcff9640 1 -- 172.21.15.64:0/2943307487 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2eb80ced50 con 0x7f2ec0106c60 2024-08-19T01:31:03.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.990+0000 7f2ebf7fe640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ea80a2b80 0x7f2ea80a5040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009710 tx=0x7f2eb0006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:03.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:02.993+0000 7f2ebcff9640 1 -- 172.21.15.64:0/2943307487 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f2eb809ba70 con 0x7f2ec0106c60 2024-08-19T01:31:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.208+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ec01a7370 con 0x7f2ec0106c60 2024-08-19T01:31:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.209+0000 7f2ebcff9640 1 -- 172.21.15.64:0/2943307487 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11848 (secure 0 0 0) 0x7f2eb809b370 con 0x7f2ec0106c60 2024-08-19T01:31:03.211 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:03.211 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"btime":"2024-08-19T01:31:01:582970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24421,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:01.216288+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24476,"qdb_cluster":[24476]},"id":1}]} 2024-08-19T01:31:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 17 2024-08-19T01:31:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ea80a2b80 msgr2=0x7f2ea80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ea80a2b80 0x7f2ea80a5040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009710 tx=0x7f2eb0006c90 comp rx=0 tx=0).stop 2024-08-19T01:31:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec0106c60 msgr2=0x7f2ec019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec0106c60 0x7f2ec019edd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800b970 tx=0x7f2eb800be30 comp rx=0 tx=0).stop 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 shutdown_connections 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ea80a2b80 0x7f2ea80a5040 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ec0106c60 0x7f2ec019edd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ec01009f0 0x7f2ec019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 --2- 172.21.15.64:0/2943307487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ec0100040 0x7f2ec019a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 >> 172.21.15.64:0/2943307487 conn(0x7f2ec0076f40 msgr2=0x7f2ec010e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 shutdown_connections 2024-08-19T01:31:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:03.211+0000 7f2ec5b98640 1 -- 172.21.15.64:0/2943307487 wait complete. 2024-08-19T01:31:03.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:03 smithi064 ceph-mon[18515]: pgmap v273: 129 pgs: 5 creating+peering, 9 unknown, 115 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:03.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:03 smithi064 ceph-mon[18515]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T01:31:03.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:03 smithi064 ceph-mon[18515]: Cluster is now healthy 2024-08-19T01:31:03.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:03 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2943307487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:03.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:03.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:03 smithi123 ceph-mon[21829]: pgmap v273: 129 pgs: 5 creating+peering, 9 unknown, 115 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:03.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:03 smithi123 ceph-mon[21829]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T01:31:03.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:03 smithi123 ceph-mon[21829]: Cluster is now healthy 2024-08-19T01:31:03.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:03 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2943307487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:03 smithi161 ceph-mon[21757]: pgmap v273: 129 pgs: 5 creating+peering, 9 unknown, 115 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:03 smithi161 ceph-mon[21757]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T01:31:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:03 smithi161 ceph-mon[21757]: Cluster is now healthy 2024-08-19T01:31:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:03 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2943307487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:04.136 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:04.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.575+0000 7fd5dc61e640 1 -- 172.21.15.64:0/1169879648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 msgr2=0x7fd5d4107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:04.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.575+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/1169879648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d4107c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009930 tx=0x7fd5d002a420 comp rx=0 tx=0).stop 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.575+0000 7fd5dc61e640 1 -- 172.21.15.64:0/1169879648 shutdown_connections 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.575+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/1169879648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5d4102250 0x7fd5d410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.575+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/1169879648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5d41018b0 0x7fd5d4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.575+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/1169879648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d4107c90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.576+0000 7fd5dc61e640 1 -- 172.21.15.64:0/1169879648 >> 172.21.15.64:0/1169879648 conn(0x7fd5d40fd5f0 msgr2=0x7fd5d40ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.576+0000 7fd5dc61e640 1 -- 172.21.15.64:0/1169879648 shutdown_connections 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.576+0000 7fd5dc61e640 1 -- 172.21.15.64:0/1169879648 wait complete. 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.576+0000 7fd5dc61e640 1 Processor -- start 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.576+0000 7fd5dc61e640 1 -- start start 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dc61e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5d41018b0 0x7fd5d4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:04.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dc61e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5d4102250 0x7fd5d41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dc61e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dc61e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4111cb0 con 0x7fd5d41078b0 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dc61e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4111b30 con 0x7fd5d4102250 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dc61e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4111e30 con 0x7fd5d41018b0 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48010/0 (socket says 172.21.15.64:48010) 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 -- 172.21.15.64:0/2339975692 learned_addr learned my addr 172.21.15.64:0/2339975692 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5da3ba640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5d41018b0 0x7fd5d4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5d9bb9640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5d4102250 0x7fd5d41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 -- 172.21.15.64:0/2339975692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5d41018b0 msgr2=0x7fd5d4198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5d41018b0 0x7fd5d4198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 -- 172.21.15.64:0/2339975692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5d4102250 msgr2=0x7fd5d41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5d4102250 0x7fd5d41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 -- 172.21.15.64:0/2339975692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d419d020 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5d9bb9640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5d4102250 0x7fd5d41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dabbb640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d419c940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc00cce0 tx=0x7fd5cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/2339975692 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc00f450 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/2339975692 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc00fad0 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d419d310 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.577+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/2339975692 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc018c10 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.578+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d41a4bf0 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.578+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d4101d10 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.580+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/2339975692 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd5cc00f5f0 con 0x7fd5d41078b0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.580+0000 7fd5cb7fe640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd5c00a2b80 0x7fd5c00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.580+0000 7fd5da3ba640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd5c00a2b80 0x7fd5c00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:04.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.580+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/2339975692 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd5cc0cfa30 con 0x7fd5d41078b0 2024-08-19T01:31:04.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.581+0000 7fd5da3ba640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd5c00a2b80 0x7fd5c00a5040 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000c00 tx=0x7fd5d0031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:04.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.583+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/2339975692 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd5cc014030 con 0x7fd5d41078b0 2024-08-19T01:31:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.760+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5d41993d0 con 0x7fd5d41078b0 2024-08-19T01:31:04.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.761+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/2339975692 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7fd5d41993d0 con 0x7fd5d41078b0 2024-08-19T01:31:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:04.763 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":89,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:01.216103+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.763+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd5c00a2b80 msgr2=0x7fd5c00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd5c00a2b80 0x7fd5c00a5040 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000c00 tx=0x7fd5d0031490 comp rx=0 tx=0).stop 2024-08-19T01:31:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 msgr2=0x7fd5d419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d419c940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc00cce0 tx=0x7fd5cc00a620 comp rx=0 tx=0).stop 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 shutdown_connections 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd5c00a2b80 0x7fd5c00a5040 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d41078b0 0x7fd5d419c940 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5d4102250 0x7fd5d41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 --2- 172.21.15.64:0/2339975692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5d41018b0 0x7fd5d4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 >> 172.21.15.64:0/2339975692 conn(0x7fd5d40fd5f0 msgr2=0x7fd5d40fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 shutdown_connections 2024-08-19T01:31:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:04.764+0000 7fd5dc61e640 1 -- 172.21.15.64:0/2339975692 wait complete. 2024-08-19T01:31:05.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:05.667 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:05.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:05 smithi064 ceph-mon[18515]: pgmap v274: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:05.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:05 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2339975692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:05.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:05 smithi123 ceph-mon[21829]: pgmap v274: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:05.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:05 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2339975692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:05 smithi161 ceph-mon[21757]: pgmap v274: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:05 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2339975692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:06.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 -- 172.21.15.64:0/3592085448 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e0102c20 msgr2=0x7f77e010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:06.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 --2- 172.21.15.64:0/3592085448 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e0102c20 0x7f77e010d770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009ca0 tx=0x7f77d802a950 comp rx=0 tx=0).stop 2024-08-19T01:31:06.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 -- 172.21.15.64:0/3592085448 shutdown_connections 2024-08-19T01:31:06.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 --2- 172.21.15.64:0/3592085448 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e0102c20 0x7f77e010d770 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 --2- 172.21.15.64:0/3592085448 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77e01022a0 0x7f77e01026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 --2- 172.21.15.64:0/3592085448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e0100b50 0x7f77e0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 -- 172.21.15.64:0/3592085448 >> 172.21.15.64:0/3592085448 conn(0x7f77e00fc500 msgr2=0x7f77e00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.132+0000 7f77e706c640 1 -- 172.21.15.64:0/3592085448 shutdown_connections 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 -- 172.21.15.64:0/3592085448 wait complete. 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 Processor -- start 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 -- start start 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77e0100b50 0x7f77e0198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e01022a0 0x7f77e0198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e0102c20 0x7f77e019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f77e0111e50 con 0x7f77e01022a0 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f77e0111cd0 con 0x7f77e0102c20 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77e706c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f77e0111fd0 con 0x7f77e0100b50 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e01022a0 0x7f77e0198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e01022a0 0x7f77e0198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48028/0 (socket says 172.21.15.64:48028) 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.133+0000 7f77dffff640 1 -- 172.21.15.64:0/570605705 learned_addr learned my addr 172.21.15.64:0/570605705 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:06.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77e4e08640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77e0100b50 0x7f77e0198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77e5609640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e0102c20 0x7f77e019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77dffff640 1 -- 172.21.15.64:0/570605705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77e0100b50 msgr2=0x7f77e0198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77dffff640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77e0100b50 0x7f77e0198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77dffff640 1 -- 172.21.15.64:0/570605705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e0102c20 msgr2=0x7f77e019cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77dffff640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e0102c20 0x7f77e019cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77dffff640 1 -- 172.21.15.64:0/570605705 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e019d2c0 con 0x7f77e01022a0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77e4e08640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77e0100b50 0x7f77e0198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77dffff640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e01022a0 0x7f77e0198850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f77d400b4a0 tx=0x7f77d400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77ddffb640 1 -- 172.21.15.64:0/570605705 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d4014070 con 0x7f77e01022a0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77ddffb640 1 -- 172.21.15.64:0/570605705 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77d4010af0 con 0x7f77e01022a0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e019d5b0 con 0x7f77e01022a0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77ddffb640 1 -- 172.21.15.64:0/570605705 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d4017d10 con 0x7f77e01022a0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.134+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e01a4e90 con 0x7f77e01022a0 2024-08-19T01:31:06.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.135+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e0100fb0 con 0x7f77e01022a0 2024-08-19T01:31:06.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.136+0000 7f77ddffb640 1 -- 172.21.15.64:0/570605705 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f77d4017070 con 0x7f77e01022a0 2024-08-19T01:31:06.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.137+0000 7f77ddffb640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f77c80a2b80 0x7f77c80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:06.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.137+0000 7f77e4e08640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f77c80a2b80 0x7f77c80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:06.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.137+0000 7f77ddffb640 1 -- 172.21.15.64:0/570605705 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f77d40ce770 con 0x7f77e01022a0 2024-08-19T01:31:06.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.137+0000 7f77e4e08640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f77c80a2b80 0x7f77c80a5040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f77d0006fd0 tx=0x7f77d0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:06.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.141+0000 7f77ddffb640 1 -- 172.21.15.64:0/570605705 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f77d409b490 con 0x7f77e01022a0 2024-08-19T01:31:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.316+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77e0108dc0 con 0x7f77e01022a0 2024-08-19T01:31:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.318+0000 7f77ddffb640 1 -- 172.21.15.64:0/570605705 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f77d409ad90 con 0x7f77e01022a0 2024-08-19T01:31:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:06.320 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":89,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:01.216103+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f77c80a2b80 msgr2=0x7f77c80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f77c80a2b80 0x7f77c80a5040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f77d0006fd0 tx=0x7f77d0008040 comp rx=0 tx=0).stop 2024-08-19T01:31:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e01022a0 msgr2=0x7f77e0198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e01022a0 0x7f77e0198850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f77d400b4a0 tx=0x7f77d400b960 comp rx=0 tx=0).stop 2024-08-19T01:31:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 shutdown_connections 2024-08-19T01:31:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f77c80a2b80 0x7f77c80a5040 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e0102c20 0x7f77e019cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77e01022a0 0x7f77e0198850 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 --2- 172.21.15.64:0/570605705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77e0100b50 0x7f77e0198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 >> 172.21.15.64:0/570605705 conn(0x7f77e00fc500 msgr2=0x7f77e00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 shutdown_connections 2024-08-19T01:31:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:06.321+0000 7f77e706c640 1 -- 172.21.15.64:0/570605705 wait complete. 2024-08-19T01:31:06.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:06 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/570605705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:06.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:06 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/570605705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:06.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:06 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/570605705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:07.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:31:07.353 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:07.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:07 smithi064 ceph-mon[18515]: pgmap v275: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:07.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 -- 172.21.15.64:0/2768811480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 msgr2=0x7f299810d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:07.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 --2- 172.21.15.64:0/2768811480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f299810d680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2990009cd0 tx=0x7f299002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:31:07.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 -- 172.21.15.64:0/2768811480 shutdown_connections 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 --2- 172.21.15.64:0/2768811480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f299810d680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 --2- 172.21.15.64:0/2768811480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29981007b0 0x7f2998100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 --2- 172.21.15.64:0/2768811480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29981067b0 0x7f2998106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 -- 172.21.15.64:0/2768811480 >> 172.21.15.64:0/2768811480 conn(0x7f29980fc4d0 msgr2=0x7f29980fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 -- 172.21.15.64:0/2768811480 shutdown_connections 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.842+0000 7f299e72f640 1 -- 172.21.15.64:0/2768811480 wait complete. 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 Processor -- start 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 -- start start 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29981007b0 0x7f2998198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:07.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f29981994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29981067b0 0x7f29981a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2998110bb0 con 0x7f2998101150 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2998110a30 con 0x7f29981007b0 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299e72f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2998110d30 con 0x7f29981067b0 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f29977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f29981994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f29977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f29981994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48040/0 (socket says 172.21.15.64:48040) 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f29977fe640 1 -- 172.21.15.64:0/281423997 learned_addr learned my addr 172.21.15.64:0/281423997 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f2997fff640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29981007b0 0x7f2998198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.843+0000 7f299cccc640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29981067b0 0x7f29981a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29977fe640 1 -- 172.21.15.64:0/281423997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29981067b0 msgr2=0x7f29981a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29977fe640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29981067b0 0x7f29981a4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29977fe640 1 -- 172.21.15.64:0/281423997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29981007b0 msgr2=0x7f2998198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29977fe640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29981007b0 0x7f2998198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:07.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29977fe640 1 -- 172.21.15.64:0/281423997 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29981a4570 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f2997fff640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29981007b0 0x7f2998198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f299cccc640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29981067b0 0x7f29981a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29977fe640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f29981994f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f298c00e9f0 tx=0x7f298c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29957fa640 1 -- 172.21.15.64:0/281423997 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c00cd80 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29957fa640 1 -- 172.21.15.64:0/281423997 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f298c010910 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29981a4740 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f29957fa640 1 -- 172.21.15.64:0/281423997 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c018990 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.844+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29981a4c80 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.845+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2998100c10 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.846+0000 7f29957fa640 1 -- 172.21.15.64:0/281423997 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f298c010ac0 con 0x7f2998101150 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.847+0000 7f29957fa640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f29800a2c00 0x7f29800a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.847+0000 7f2997fff640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f29800a2c00 0x7f29800a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:07.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.847+0000 7f29957fa640 1 -- 172.21.15.64:0/281423997 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f298c0cf650 con 0x7f2998101150 2024-08-19T01:31:07.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.847+0000 7f2997fff640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f29800a2c00 0x7f29800a50c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2988005fd0 tx=0x7f2988005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:07.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:07.851+0000 7f29957fa640 1 -- 172.21.15.64:0/281423997 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f298c09c370 con 0x7f2998101150 2024-08-19T01:31:07.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:07 smithi123 ceph-mon[21829]: pgmap v275: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:07.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:07 smithi161 ceph-mon[21757]: pgmap v275: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:08.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.062+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2998108d40 con 0x7f2998101150 2024-08-19T01:31:08.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.062+0000 7f29957fa640 1 -- 172.21.15.64:0/281423997 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11848 (secure 0 0 0) 0x7f298c09bc70 con 0x7f2998101150 2024-08-19T01:31:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:08.064 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"btime":"2024-08-19T01:31:01:582970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24415,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24421,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:01.216288+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24476,"qdb_cluster":[24476]},"id":1}]} 2024-08-19T01:31:08.064 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 17 2024-08-19T01:31:08.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f29800a2c00 msgr2=0x7f29800a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f29800a2c00 0x7f29800a50c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2988005fd0 tx=0x7f2988005850 comp rx=0 tx=0).stop 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 msgr2=0x7f29981994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f29981994f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f298c00e9f0 tx=0x7f298c00eeb0 comp rx=0 tx=0).stop 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 shutdown_connections 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f29800a2c00 0x7f29800a50c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29981067b0 0x7f29981a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2998101150 0x7f29981994f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 --2- 172.21.15.64:0/281423997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29981007b0 0x7f2998198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 >> 172.21.15.64:0/281423997 conn(0x7f29980fc4d0 msgr2=0x7f29980fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 shutdown_connections 2024-08-19T01:31:08.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:08.065+0000 7f299e72f640 1 -- 172.21.15.64:0/281423997 wait complete. 2024-08-19T01:31:08.713 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:08.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:08 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/281423997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:08.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:08 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/281423997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:08.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:08 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/281423997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:09.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/652129763 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 msgr2=0x7f9fd0106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:09.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/652129763 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd0106b90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0009cd0 tx=0x7f9fc002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/652129763 shutdown_connections 2024-08-19T01:31:09.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/652129763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd0101150 0x7f9fd010d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/652129763 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fd01007b0 0x7f9fd0100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/652129763 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd0106b90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/652129763 >> 172.21.15.64:0/652129763 conn(0x7f9fd00fc4d0 msgr2=0x7f9fd00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:09.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/652129763 shutdown_connections 2024-08-19T01:31:09.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.425+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/652129763 wait complete. 2024-08-19T01:31:09.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 Processor -- start 2024-08-19T01:31:09.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 -- start start 2024-08-19T01:31:09.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fd01007b0 0x7f9fd0197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:09.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd0101150 0x7f9fd01975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:09.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:09.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0110bd0 con 0x7f9fd01067b0 2024-08-19T01:31:09.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0110a50 con 0x7f9fd0101150 2024-08-19T01:31:09.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fd5cf0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0110d50 con 0x7f9fd01007b0 2024-08-19T01:31:09.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fcffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:09.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fcffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd019b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54428/0 (socket says 172.21.15.64:54428) 2024-08-19T01:31:09.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fcffff640 1 -- 172.21.15.64:0/897153583 learned_addr learned my addr 172.21.15.64:0/897153583 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:09.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.426+0000 7f9fcf7fe640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fd01007b0 0x7f9fd0197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:09.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fceffd640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd0101150 0x7f9fd01975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:09.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fcffff640 1 -- 172.21.15.64:0/897153583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fd01007b0 msgr2=0x7f9fd0197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:09.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fcffff640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fd01007b0 0x7f9fd0197070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fcffff640 1 -- 172.21.15.64:0/897153583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd0101150 msgr2=0x7f9fd01975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:09.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fcffff640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd0101150 0x7f9fd01975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fcffff640 1 -- 172.21.15.64:0/897153583 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd019c020 con 0x7f9fd01067b0 2024-08-19T01:31:09.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fcf7fe640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fd01007b0 0x7f9fd0197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fceffd640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd0101150 0x7f9fd01975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fcffff640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd019b940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009fd0 tx=0x7f9fc800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fccff9640 1 -- 172.21.15.64:0/897153583 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8011830 con 0x7f9fd01067b0 2024-08-19T01:31:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd019c310 con 0x7f9fd01067b0 2024-08-19T01:31:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fccff9640 1 -- 172.21.15.64:0/897153583 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc8016070 con 0x7f9fd01067b0 2024-08-19T01:31:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fccff9640 1 -- 172.21.15.64:0/897153583 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc800f2e0 con 0x7f9fd01067b0 2024-08-19T01:31:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.427+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd01a3bf0 con 0x7f9fd01067b0 2024-08-19T01:31:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.428+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd0100c10 con 0x7f9fd01067b0 2024-08-19T01:31:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.429+0000 7f9fccff9640 1 -- 172.21.15.64:0/897153583 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9fc80119d0 con 0x7f9fd01067b0 2024-08-19T01:31:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.430+0000 7f9fccff9640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9fb80a2b80 0x7f9fb80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.430+0000 7f9fccff9640 1 -- 172.21.15.64:0/897153583 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9fc80ce9b0 con 0x7f9fd01067b0 2024-08-19T01:31:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.430+0000 7f9fcf7fe640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9fb80a2b80 0x7f9fb80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.430+0000 7f9fcf7fe640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9fb80a2b80 0x7f9fb80a5040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000f80 tx=0x7f9fc0009890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.434+0000 7f9fccff9640 1 -- 172.21.15.64:0/897153583 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f9fc809b6d0 con 0x7f9fd01067b0 2024-08-19T01:31:09.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.611+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9fd01a3e20 con 0x7f9fd01067b0 2024-08-19T01:31:09.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.612+0000 7f9fccff9640 1 -- 172.21.15.64:0/897153583 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f9fc809afd0 con 0x7f9fd01067b0 2024-08-19T01:31:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:09.614 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":89,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:01.216103+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:09.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9fb80a2b80 msgr2=0x7f9fb80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:09.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9fb80a2b80 0x7f9fb80a5040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000f80 tx=0x7f9fc0009890 comp rx=0 tx=0).stop 2024-08-19T01:31:09.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 msgr2=0x7f9fd019b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:09.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd019b940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009fd0 tx=0x7f9fc800b8e0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 shutdown_connections 2024-08-19T01:31:09.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9fb80a2b80 0x7f9fb80a5040 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fd01067b0 0x7f9fd019b940 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd0101150 0x7f9fd01975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 --2- 172.21.15.64:0/897153583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fd01007b0 0x7f9fd0197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:09.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 >> 172.21.15.64:0/897153583 conn(0x7f9fd00fc4d0 msgr2=0x7f9fd00fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:09.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 shutdown_connections 2024-08-19T01:31:09.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:09.615+0000 7f9fd5cf0640 1 -- 172.21.15.64:0/897153583 wait complete. 2024-08-19T01:31:09.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:09 smithi123 ceph-mon[21829]: pgmap v276: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:09.934 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:09 smithi064 ceph-mon[18515]: pgmap v276: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:09 smithi161 ceph-mon[21757]: pgmap v276: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:10.316 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-19T01:30:52.282253+0000', 'modified': '2024-08-19T01:31:01.216288+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24476}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24476': {'gid': 24476, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/2281975354', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 2281975354}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 2281975354}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24476, 'qdb_cluster': [24476]} 2024-08-19T01:31:10.316 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-19T01:31:10.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.b status 2024-08-19T01:31:10.578 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:10.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:10 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/897153583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:10.947 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:10 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/897153583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:10.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:10 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/897153583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:11.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 -- 172.21.15.64:0/1893950875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41067b0 msgr2=0x7f38e4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 --2- 172.21.15.64:0/1893950875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41067b0 0x7f38e4106b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009930 tx=0x7f38d402a420 comp rx=0 tx=0).stop 2024-08-19T01:31:11.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 -- 172.21.15.64:0/1893950875 shutdown_connections 2024-08-19T01:31:11.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 --2- 172.21.15.64:0/1893950875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38e4101150 0x7f38e410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 --2- 172.21.15.64:0/1893950875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e41007b0 0x7f38e4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 --2- 172.21.15.64:0/1893950875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41067b0 0x7f38e4106b90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 -- 172.21.15.64:0/1893950875 >> 172.21.15.64:0/1893950875 conn(0x7f38e40fc4d0 msgr2=0x7f38e40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 -- 172.21.15.64:0/1893950875 shutdown_connections 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.028+0000 7f38e9907640 1 -- 172.21.15.64:0/1893950875 wait complete. 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 Processor -- start 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 -- start start 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41007b0 0x7f38e4198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e4101150 0x7f38e41994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38e41067b0 0x7f38e41a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f38e4110bd0 con 0x7f38e41007b0 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38e4110a50 con 0x7f38e4101150 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e9907640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f38e4110d50 con 0x7f38e41067b0 2024-08-19T01:31:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41007b0 0x7f38e4198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41007b0 0x7f38e4198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54452/0 (socket says 172.21.15.64:54452) 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3323157065 learned_addr learned my addr 172.21.15.64:0/3323157065 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3323157065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38e41067b0 msgr2=0x7f38e41a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38e41067b0 0x7f38e41a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38dbfff640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e4101150 0x7f38e41994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3323157065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e4101150 msgr2=0x7f38e41994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e4101150 0x7f38e41994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.029+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3323157065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e41a4520 con 0x7f38e41007b0 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.030+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41007b0 0x7f38e4198f60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009a50 tx=0x7f38d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.030+0000 7f38e8905640 1 -- 172.21.15.64:0/3323157065 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d403f030 con 0x7f38e41007b0 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.030+0000 7f38e8905640 1 -- 172.21.15.64:0/3323157065 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38d4003ce0 con 0x7f38e41007b0 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.030+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e41a46f0 con 0x7f38e41007b0 2024-08-19T01:31:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.030+0000 7f38e8905640 1 -- 172.21.15.64:0/3323157065 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4035070 con 0x7f38e41007b0 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.030+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e41a4bd0 con 0x7f38e41007b0 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38e8905640 1 -- 172.21.15.64:0/3323157065 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f38d4042030 con 0x7f38e41007b0 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 Processor -- start 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 -- start start 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0d12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0d3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5623dc02b710 con 0x5623dc0c6f50 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623dbfc3770 con 0x5623dc0c6630 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.032+0000 7f38ebd05740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0276d0 con 0x5623dc0d1810 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0d12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0d12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54466/0 (socket says 172.21.15.64:54466) 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 -- 172.21.15.64:0/4247629842 learned_addr learned my addr 172.21.15.64:0/4247629842 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38dbfff640 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0d3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e37fe640 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 -- 172.21.15.64:0/4247629842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 msgr2=0x5623dc0d3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0d3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 -- 172.21.15.64:0/4247629842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 msgr2=0x5623dc0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e8905640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f38d00a2c00 0x7f38d00a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 -- 172.21.15.64:0/4247629842 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623dbfca3e0 con 0x5623dc0c6f50 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38dbfff640 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0d3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e37fe640 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e37fe640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f38d00a2c00 0x7f38d00a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e8905640 1 -- 172.21.15.64:0/3323157065 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f38d40ef8e0 con 0x7f38e41007b0 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0d12d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f38d40b8360 tx=0x7f38d40b8260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733153a88a13269c server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.033+0000 7f38d9ffb640 1 -- 172.21.15.64:0/4247629842 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4049070 con 0x5623dc0c6f50 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38d9ffb640 1 -- 172.21.15.64:0/4247629842 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38d40bc600 con 0x5623dc0c6f50 2024-08-19T01:31:11.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38d9ffb640 1 -- 172.21.15.64:0/4247629842 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40bbcd0 con 0x5623dc0c6f50 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 -- 172.21.15.64:0/4247629842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 msgr2=0x5623dc0d12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38e37fe640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f38d00a2c00 0x7f38d00a50c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009e80 tx=0x7f38dc009430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0d12d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f38d40b8360 tx=0x7f38d40b8260 comp rx=0 tx=0).stop 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 -- 172.21.15.64:0/4247629842 shutdown_connections 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0d3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0d12d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 --2- 172.21.15.64:0/4247629842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 -- 172.21.15.64:0/4247629842 >> 172.21.15.64:0/4247629842 conn(0x5623dc0c2bc0 msgr2=0x5623dc0c2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 -- 172.21.15.64:0/4247629842 shutdown_connections 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.034+0000 7f38ebd05740 1 -- 172.21.15.64:0/4247629842 wait complete. 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 -- start start 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 Processor -- start 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0d41a0 con 0x5623dc0c6f50 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623dbfe6970 con 0x5623dc0c6630 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0d4320 con 0x5623dc0d1810 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38ebd05740 10 client.0 _ll_register_callbacks cb 0x5623dbd3a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54468/0 (socket says 172.21.15.64:54468) 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38dbfff640 1 -- 172.21.15.64:0/2186115887 learned_addr learned my addr 172.21.15.64:0/2186115887 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38dbfff640 1 -- 172.21.15.64:0/2186115887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 msgr2=0x5623dc0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38dbfff640 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38dbfff640 1 -- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 msgr2=0x5623dc0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.035+0000 7f38dbfff640 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38dbfff640 1 -- 172.21.15.64:0/2186115887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623dbd9d2b0 con 0x5623dc0c6f50 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38dbfff640 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0cec50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00f6f0 tx=0x7f38cc00fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6731fc9668b5b91a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38c2ffd640 1 -- 172.21.15.64:0/2186115887 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc016070 con 0x5623dc0c6f50 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d40bc600 con 0x5623dc0c6f50 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38c2ffd640 1 -- 172.21.15.64:0/2186115887 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f38cc00baa0 con 0x5623dc0c6f50 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38c2ffd640 1 -- 172.21.15.64:0/2186115887 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f38cc00bca0 con 0x5623dc0c6f50 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 10 client.14730 fetch_fsmap learned FSMap version 17 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5623dc0dfd70 con 0x5623dc0c6f50 2024-08-19T01:31:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38c2ffd640 1 -- 172.21.15.64:0/2186115887 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38d40bc600 con 0x5623dc0c6f50 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38c2ffd640 1 -- 172.21.15.64:0/2186115887 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f38cc0193b0 con 0x5623dc0c6f50 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 10 client.14730 fetch_fsmap finished waiting for FSMap version 17 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 10 client.14730 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 20 client.14730 populate_metadata read hostname 'smithi064' 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc0e2470 0x5623dc0e38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.036+0000 7f38ebd05740 4 client.14730 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.037+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623dc038c80 con 0x5623dc0e2470 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.037+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc0e2470 0x5623dc0e38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.037+0000 7f38c2ffd640 10 client.14730 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.037+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc0e2470 0x5623dc0e38e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f4e1ac46447a23f server_cookie=2f50f4a69a989f76 in_seq=0 out_seq=0 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38c2ffd640 1 -- 172.21.15.64:0/2186115887 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5623dc038c80 con 0x5623dc0e2470 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38c2ffd640 10 client.14730 handle_command_reply: tid=0 2024-08-19T01:31:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 client.14730 shutdown 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 msgr2=0x5623dc0cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0cec50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00f6f0 tx=0x7f38cc00fbb0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 shutdown_connections 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc0cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6f50 0x5623dc0cec50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc0e2470 0x5623dc0e38e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 --2- 172.21.15.64:0/2186115887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6630 0x5623dc0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38c2ffd640 0 client.14730 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 >> 172.21.15.64:0/2186115887 conn(0x5623dc0c2bc0 msgr2=0x5623dc0c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 shutdown_connections 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.038+0000 7f38ebd05740 1 -- 172.21.15.64:0/2186115887 wait complete. 2024-08-19T01:31:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.039+0000 7f38ebd05740 20 client.14730 trim_cache size 0 max 16384 2024-08-19T01:31:11.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.056+0000 7f38ebd05740 1 Processor -- start 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.056+0000 7f38ebd05740 1 -- start start 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.056+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.056+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0c6f50 0x5623dc0e2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0d1810 0x5623dc0dcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38ebd05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0e1c20 con 0x5623dc0c6630 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38ebd05740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0e22f0 con 0x5623dc0d1810 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38ebd05740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0e29c0 con 0x5623dc0c6f50 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0d0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54484/0 (socket says 172.21.15.64:54484) 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 -- 172.21.15.64:0/1328613600 learned_addr learned my addr 172.21.15.64:0/1328613600 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38dbfff640 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0c6f50 0x5623dc0e2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e37fe640 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0d1810 0x5623dc0dcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 -- 172.21.15.64:0/1328613600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0c6f50 msgr2=0x5623dc0e2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0c6f50 0x5623dc0e2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 -- 172.21.15.64:0/1328613600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0d1810 msgr2=0x5623dc0dcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0d1810 0x5623dc0dcd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 -- 172.21.15.64:0/1328613600 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623dc0c29f0 con 0x5623dc0c6630 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38dbfff640 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0c6f50 0x5623dc0e2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e37fe640 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0d1810 0x5623dc0dcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0d0e70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ed510 tx=0x7f38d40f5630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd64920db5f7609 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38d97fa640 1 -- 172.21.15.64:0/1328613600 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4049040 con 0x5623dc0c6630 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38d97fa640 1 -- 172.21.15.64:0/1328613600 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38d40f5400 con 0x5623dc0c6630 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.057+0000 7f38d97fa640 1 -- 172.21.15.64:0/1328613600 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40bb860 con 0x5623dc0c6630 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 -- 172.21.15.64:0/1328613600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 msgr2=0x5623dc0d0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0d0e70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ed510 tx=0x7f38d40f5630 comp rx=0 tx=0).stop 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 -- 172.21.15.64:0/1328613600 shutdown_connections 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0d1810 0x5623dc0dcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0c6f50 0x5623dc0e2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 --2- 172.21.15.64:0/1328613600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0d0e70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 -- 172.21.15.64:0/1328613600 >> 172.21.15.64:0/1328613600 conn(0x5623dc0c2bc0 msgr2=0x5623dc0ccfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 -- 172.21.15.64:0/1328613600 shutdown_connections 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 -- 172.21.15.64:0/1328613600 wait complete. 2024-08-19T01:31:11.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.058+0000 7f38ebd05740 1 -- start start 2024-08-19T01:31:11.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 1 Processor -- start 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6f50 0x5623dc0e5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc14ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0e3760 con 0x5623dc0c6630 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0e3090 con 0x5623dc0c6f50 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623dc0e3e30 con 0x5623dc0d1810 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38ebd05740 10 client.0 _ll_register_callbacks cb 0x7f38cc005f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0e4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54488/0 (socket says 172.21.15.64:54488) 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3428244290 learned_addr learned my addr 172.21.15.64:0/3428244290 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:11.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38dbfff640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6f50 0x5623dc0e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3428244290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 msgr2=0x5623dc14ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e37fe640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc14ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc14ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6f50 msgr2=0x5623dc0e5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6f50 0x5623dc0e5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 -- 172.21.15.64:0/3428244290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623dc0be130 con 0x5623dc0c6630 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38dbfff640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6f50 0x5623dc0e5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38e2ffd640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0e4c50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f38d40bbc00 tx=0x7f38d403b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18ca8bccd0440f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.059+0000 7f38c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38c2ffd640 1 -- 172.21.15.64:0/3428244290 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d40bba40 con 0x5623dc0c6630 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38c2ffd640 1 -- 172.21.15.64:0/3428244290 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f38d40f5810 con 0x5623dc0c6630 2024-08-19T01:31:11.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d40f5400 con 0x5623dc0c6630 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38c2ffd640 1 -- 172.21.15.64:0/3428244290 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f38d40bb860 con 0x5623dc0c6630 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 10 client.14736 fetch_fsmap learned FSMap version 17 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5623dc0be130 con 0x5623dc0c6630 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38c2ffd640 1 -- 172.21.15.64:0/3428244290 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38d40f5400 con 0x5623dc0c6630 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38c2ffd640 1 -- 172.21.15.64:0/3428244290 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f38d40da020 con 0x5623dc0c6630 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 10 client.14736 fetch_fsmap finished waiting for FSMap version 17 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 10 client.14736 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 20 client.14736 populate_metadata read hostname 'smithi064' 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc1504c0 0x5623dc14f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 4 client.14736 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "status"}) -- 0x5623dc033190 con 0x5623dc1504c0 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.060+0000 7f38dbfff640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc1504c0 0x5623dc14f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38dbfff640 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc1504c0 0x5623dc14f1d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d621d205b44333df server_cookie=977d091745ce0e3a in_seq=0 out_seq=0 2024-08-19T01:31:11.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38c2ffd640 10 client.14736 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38c2ffd640 1 -- 172.21.15.64:0/3428244290 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5623dc033190 con 0x5623dc1504c0 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38c2ffd640 10 client.14736 handle_command_reply: tid=0 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 client.14736 shutdown 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 msgr2=0x5623dc0e4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0e4c50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f38d40bbc00 tx=0x7f38d403b790 comp rx=0 tx=0).stop 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 shutdown_connections 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623dc0d1810 0x5623dc14ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5623dc1504c0 0x5623dc14f1d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623dc0c6f50 0x5623dc0e5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 --2- 172.21.15.64:0/3428244290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5623dc0c6630 0x5623dc0e4c50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 >> 172.21.15.64:0/3428244290 conn(0x5623dc0c2bc0 msgr2=0x5623dc0cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38c2ffd640 0 client.14736 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:31:11.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 shutdown_connections 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 1 -- 172.21.15.64:0/3428244290 wait complete. 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.061+0000 7f38ebd05740 20 client.14736 trim_cache size 0 max 16384 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24476, 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:31:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 18.515403388999999, 2024-08-19T01:31:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 17, 2024-08-19T01:31:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 89, 2024-08-19T01:31:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 88, 2024-08-19T01:31:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 29.522476298000001 2024-08-19T01:31:11.071 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:31:11.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f38d00a2c00 msgr2=0x7f38d00a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f38d00a2c00 0x7f38d00a50c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009e80 tx=0x7f38dc009430 comp rx=0 tx=0).stop 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41007b0 msgr2=0x7f38e4198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41007b0 0x7f38e4198f60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009a50 tx=0x7f38d4004270 comp rx=0 tx=0).stop 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 shutdown_connections 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f38d00a2c00 0x7f38d00a50c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38e41067b0 0x7f38e41a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e4101150 0x7f38e41994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 --2- 172.21.15.64:0/3323157065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38e41007b0 0x7f38e4198f60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 >> 172.21.15.64:0/3323157065 conn(0x7f38e40fc4d0 msgr2=0x7f38e40fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 shutdown_connections 2024-08-19T01:31:11.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:11.062+0000 7f38e9907640 1 -- 172.21.15.64:0/3323157065 wait complete. 2024-08-19T01:31:11.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:11 smithi064 ceph-mon[18515]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:11.777 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-19T01:31:11.777 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:11.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs ls 2024-08-19T01:31:11.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:11 smithi123 ceph-mon[21829]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:11 smithi161 ceph-mon[21757]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:12.034 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 -- 172.21.15.64:0/1857172959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 msgr2=0x7f65e0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/1857172959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0101d10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009930 tx=0x7f65d002a420 comp rx=0 tx=0).stop 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 -- 172.21.15.64:0/1857172959 shutdown_connections 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/1857172959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65e0102250 0x7f65e010e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/1857172959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0101d10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/1857172959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e01078b0 0x7f65e0107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 -- 172.21.15.64:0/1857172959 >> 172.21.15.64:0/1857172959 conn(0x7f65e00fd5b0 msgr2=0x7f65e00ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 -- 172.21.15.64:0/1857172959 shutdown_connections 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 -- 172.21.15.64:0/1857172959 wait complete. 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 Processor -- start 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.517+0000 7f65e7a5f640 1 -- start start 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e7a5f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e7a5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65e0102250 0x7f65e0198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e7a5f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e01078b0 0x7f65e019cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e7a5f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f65e0111cd0 con 0x7f65e01018b0 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e7a5f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65e0111b50 con 0x7f65e01078b0 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e7a5f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f65e0111e50 con 0x7f65e0102250 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54504/0 (socket says 172.21.15.64:54504) 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 -- 172.21.15.64:0/3319768330 learned_addr learned my addr 172.21.15.64:0/3319768330 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e5ffc640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e01078b0 0x7f65e019cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e4ffa640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65e0102250 0x7f65e0198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 -- 172.21.15.64:0/3319768330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65e0102250 msgr2=0x7f65e0198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65e0102250 0x7f65e0198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 -- 172.21.15.64:0/3319768330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e01078b0 msgr2=0x7f65e019cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e01078b0 0x7f65e019cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 -- 172.21.15.64:0/3319768330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e019d1e0 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e4ffa640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65e0102250 0x7f65e0198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e57fb640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0198230 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00b4a0 tx=0x7f65dc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65d67fc640 1 -- 172.21.15.64:0/3319768330 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc014070 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e019d4d0 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65d67fc640 1 -- 172.21.15.64:0/3319768330 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65dc010af0 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.518+0000 7f65d67fc640 1 -- 172.21.15.64:0/3319768330 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc017d10 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.519+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e01a4db0 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.520+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e0101d10 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.521+0000 7f65d67fc640 1 -- 172.21.15.64:0/3319768330 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f65dc017070 con 0x7f65e01018b0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.521+0000 7f65d67fc640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f65cc0a2b80 0x7f65cc0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.522+0000 7f65e4ffa640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f65cc0a2b80 0x7f65cc0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.522+0000 7f65d67fc640 1 -- 172.21.15.64:0/3319768330 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f65dc0ce9e0 con 0x7f65e01018b0 2024-08-19T01:31:12.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.522+0000 7f65e4ffa640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f65cc0a2b80 0x7f65cc0a5040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f65d0000c00 tx=0x7f65d0032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:12.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.525+0000 7f65d67fc640 1 -- 172.21.15.64:0/3319768330 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f65dc09b700 con 0x7f65e01018b0 2024-08-19T01:31:12.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.739+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f65e0199590 con 0x7f65e01018b0 2024-08-19T01:31:12.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.739+0000 7f65d67fc640 1 -- 172.21.15.64:0/3319768330 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f65e0199590 con 0x7f65e01018b0 2024-08-19T01:31:12.740 INFO:teuthology.orchestra.run.smithi064.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-08-19T01:31:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f65cc0a2b80 msgr2=0x7f65cc0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f65cc0a2b80 0x7f65cc0a5040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f65d0000c00 tx=0x7f65d0032d50 comp rx=0 tx=0).stop 2024-08-19T01:31:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 msgr2=0x7f65e0198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0198230 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00b4a0 tx=0x7f65dc00b960 comp rx=0 tx=0).stop 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 shutdown_connections 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f65cc0a2b80 0x7f65cc0a5040 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e01078b0 0x7f65e019cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65e0102250 0x7f65e0198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 --2- 172.21.15.64:0/3319768330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65e01018b0 0x7f65e0198230 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 >> 172.21.15.64:0/3319768330 conn(0x7f65e00fd5b0 msgr2=0x7f65e00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 shutdown_connections 2024-08-19T01:31:12.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:12.741+0000 7f65e7a5f640 1 -- 172.21.15.64:0/3319768330 wait complete. 2024-08-19T01:31:13.462 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-19T01:31:13.462 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-19T01:31:13.462 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi064.front.sepia.ceph.com 2024-08-19T01:31:13.462 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-08-19T01:31:13.462 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:13.463 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-19T01:31:13.463 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-19T01:31:13.463 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-19T01:31:13.465 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-08-19T01:31:13.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:13.466 DEBUG:teuthology.orchestra.run.smithi064:> ip addr 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: inet 127.0.0.1/8 scope host lo 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 ::1/128 scope host 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:78 brd ff:ff:ff:ff:ff:ff 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f0 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-19T01:31:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:79 brd ff:ff:ff:ff:ff:ff 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f1 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:de brd ff:ff:ff:ff:ff:ff 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:df brd ff:ff:ff:ff:ff:ff 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: inet 172.21.15.64/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft 42403sec preferred_lft 42403sec 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 fe80::ec4:7aff:feb6:f8df/64 scope link noprefixroute 2024-08-19T01:31:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-08-19T01:31:13.484 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-08-19T01:31:13.484 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:31:13.484 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:31:13.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add name ceph-brx type bridge 2024-08-19T01:31:13.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr flush dev ceph-brx 2024-08-19T01:31:13.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set ceph-brx up 2024-08-19T01:31:13.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-08-19T01:31:13.484 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:31:13.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:31:13.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:31:13.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:13.684 DEBUG:teuthology.orchestra.run.smithi064:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-19T01:31:13.690 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:13 smithi064 ceph-mon[18515]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:13.690 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:13 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3319768330' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-08-19T01:31:13.710 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-08-19T01:31:13.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:13.711 DEBUG:teuthology.orchestra.run.smithi064:> route 2024-08-19T01:31:13.779 INFO:teuthology.orchestra.run.smithi064.stdout:Kernel IP routing table 2024-08-19T01:31:13.780 INFO:teuthology.orchestra.run.smithi064.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-19T01:31:13.780 INFO:teuthology.orchestra.run.smithi064.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-08-19T01:31:13.780 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-08-19T01:31:13.780 INFO:teuthology.orchestra.run.smithi064.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-19T01:31:13.780 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:31:13.780 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:31:13.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-08-19T01:31:13.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-08-19T01:31:13.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-08-19T01:31:13.781 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:31:13.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:31:13.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:13 smithi123 ceph-mon[21829]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:13 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3319768330' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-08-19T01:31:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:13 smithi161 ceph-mon[21757]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:13 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3319768330' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-08-19T01:31:14.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:31:14.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:14.074 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-08-19T01:31:14.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:14.089 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list-id 2024-08-19T01:31:14.145 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:31:14.145 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:31:14.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-19T01:31:14.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-08-19T01:31:14.145 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:31:14.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:31:14.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:31:14.262 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-08-19T01:31:14.262 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:31:14.263 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:31:14.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-08-19T01:31:14.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-08-19T01:31:14.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-08-19T01:31:14.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-08-19T01:31:14.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-08-19T01:31:14.263 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:31:14.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:31:14.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:31:14.401 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:31:14.401 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:31:14.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set brx.0 up 2024-08-19T01:31:14.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set dev brx.0 master ceph-brx 2024-08-19T01:31:14.401 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:31:14.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:31:14.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:31:14.477 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-08-19T01:31:14.478 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-19T01:31:14.478 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:14.492 INFO:teuthology.orchestra.run.smithi064.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-08-19T01:31:14.492 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-19T01:31:14.493 DEBUG:teuthology.orchestra.run.smithi064:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:14.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2024-08-19T01:31:14.613 DEBUG:teuthology.orchestra.run.smithi064:> cat /proc/self/mounts | awk '{print $2}' 2024-08-19T01:31:14.669 INFO:teuthology.orchestra.run.smithi064.stdout:/proc 2024-08-19T01:31:14.669 INFO:teuthology.orchestra.run.smithi064.stdout:/sys 2024-08-19T01:31:14.669 INFO:teuthology.orchestra.run.smithi064.stdout:/dev 2024-08-19T01:31:14.669 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/security 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/shm 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/pts 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/run 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/cgroup 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/pstore 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/bpf 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/ 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/selinux 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/proc/sys/fs/binfmt_misc 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/mqueue 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/hugepages 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/debug 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/tracing 2024-08-19T01:31:14.670 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-sysusers.service 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/config 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/fuse/connections 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-sysctl.service 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/nfs/rpc_pipefs 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/run/user/1000 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/ceph 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/78a61ea90060f2fb1a97da9c023aef32b4b16f02f1a52c679c9dea579cbafe5b/merged 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/bc4631c055e41ca0ff417a5df8ed64e1cf21283440b39b009ca70cb284ad7731/merged 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/run/user/0 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/proc/sys/fs/binfmt_misc 2024-08-19T01:31:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/62640d5d5be2df76b219402bbd6545abb19cc030c1bd4277df46f4ecde58a99f/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/0b7750f64ed6891f1ec940ee41439b007be04c9d0c2168860481ecab01bbcb57/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/7eea355943ac5f7f87a3b00af743a2ea6d2ddf493931a4bb13003bbad5f346e3/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/f2e72fd487e045de95c8096d0a8c0cf53aacf697d94613f885db5c451a4641cf/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/e6069ec0184c1b35075ef81ce439226562aa2acf2ebd21eec54d7a017223de68/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/333e4ecca7d96d9d37eda2805138f87c082829c323fd8d3724cda298b88407ed/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/240004c6d2c2c7d6a72c7866ef9787032ba199ba206a5ec99507d3d91a379c0c/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/2cbcca884937cfa1dd1652ef0d7fb6c08cfcee36fa6dcf4b4689e3b6229742f8/merged 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/run/netns 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run.smithi064.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-19T01:31:14.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:14.673 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2024-08-19T01:31:14.724 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-08-19T01:31:14.724 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-08-19T01:31:14.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2024-08-19T01:31:14.790 DEBUG:teuthology.orchestra.run.smithi064:> cat /proc/self/mounts | awk '{print $2}' 2024-08-19T01:31:14.846 INFO:teuthology.orchestra.run.smithi064.stdout:/proc 2024-08-19T01:31:14.846 INFO:teuthology.orchestra.run.smithi064.stdout:/sys 2024-08-19T01:31:14.846 INFO:teuthology.orchestra.run.smithi064.stdout:/dev 2024-08-19T01:31:14.846 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/security 2024-08-19T01:31:14.846 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/shm 2024-08-19T01:31:14.846 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/pts 2024-08-19T01:31:14.846 INFO:teuthology.orchestra.run.smithi064.stdout:/run 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/cgroup 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/pstore 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/bpf 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/ 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/selinux 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/proc/sys/fs/binfmt_misc 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/mqueue 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/hugepages 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/debug 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/tracing 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-sysusers.service 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/kernel/config 2024-08-19T01:31:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:/sys/fs/fuse/connections 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-sysctl.service 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/nfs/rpc_pipefs 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/run/user/1000 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/ceph 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/78a61ea90060f2fb1a97da9c023aef32b4b16f02f1a52c679c9dea579cbafe5b/merged 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/bc4631c055e41ca0ff417a5df8ed64e1cf21283440b39b009ca70cb284ad7731/merged 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/run/user/0 2024-08-19T01:31:14.848 INFO:teuthology.orchestra.run.smithi064.stdout:/proc/sys/fs/binfmt_misc 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/62640d5d5be2df76b219402bbd6545abb19cc030c1bd4277df46f4ecde58a99f/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/0b7750f64ed6891f1ec940ee41439b007be04c9d0c2168860481ecab01bbcb57/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/7eea355943ac5f7f87a3b00af743a2ea6d2ddf493931a4bb13003bbad5f346e3/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/f2e72fd487e045de95c8096d0a8c0cf53aacf697d94613f885db5c451a4641cf/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/e6069ec0184c1b35075ef81ce439226562aa2acf2ebd21eec54d7a017223de68/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/333e4ecca7d96d9d37eda2805138f87c082829c323fd8d3724cda298b88407ed/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/240004c6d2c2c7d6a72c7866ef9787032ba199ba206a5ec99507d3d91a379c0c/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/var/lib/containers/storage/overlay/2cbcca884937cfa1dd1652ef0d7fb6c08cfcee36fa6dcf4b4689e3b6229742f8/merged 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/run/netns 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-19T01:31:14.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:14.850 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2024-08-19T01:31:14.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:2024-08-19T01:31:14.866+0000 7fcd31d10480 -1 init, newargv = 0x563c4561d980 newargc=15 2024-08-19T01:31:14.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[59845]: starting ceph client 2024-08-19T01:31:14.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[59845]: starting fuse 2024-08-19T01:31:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:121 2024-08-19T01:31:14.883 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2024-08-19T01:31:14.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo stdin-killer -- python3 -c ' 2024-08-19T01:31:14.883 DEBUG:teuthology.orchestra.run.smithi064:> import glob 2024-08-19T01:31:14.883 DEBUG:teuthology.orchestra.run.smithi064:> import re 2024-08-19T01:31:14.883 DEBUG:teuthology.orchestra.run.smithi064:> import os 2024-08-19T01:31:14.883 DEBUG:teuthology.orchestra.run.smithi064:> import subprocess 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> def _find_admin_socket(client_name): 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> files = glob.glob(asok_path) 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> # Given a non-glob path, it better be there 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> if "*" not in asok_path: 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> assert(len(files) == 1) 2024-08-19T01:31:14.884 DEBUG:teuthology.orchestra.run.smithi064:> return files[0] 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> for f in files: 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> if os.path.exists("/proc/{0}".format(pid)): 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> contents = proc_f.read() 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> if mountpoint in contents: 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> return f 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> print(_find_admin_socket("client.admin")) 2024-08-19T01:31:14.885 DEBUG:teuthology.orchestra.run.smithi064:> ' 2024-08-19T01:31:15.001 INFO:teuthology.orchestra.run.smithi064.stdout:/var/run/ceph/ceph-client.admin.59845.asok 2024-08-19T01:31:15.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:31:15.012 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.59845.asok 2024-08-19T01:31:15.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:15.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.59845.asok status 2024-08-19T01:31:15.147 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:31:15.147 INFO:teuthology.orchestra.run.smithi064.stdout: "metadata": { 2024-08-19T01:31:15.147 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_sha1": "de09dd018de4baddfa6ca4d43bc8fb784cf75833", 2024-08-19T01:31:15.147 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc)", 2024-08-19T01:31:15.147 INFO:teuthology.orchestra.run.smithi064.stdout: "entity_id": "admin", 2024-08-19T01:31:15.147 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": "smithi064", 2024-08-19T01:31:15.147 INFO:teuthology.orchestra.run.smithi064.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "pid": "59845", 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "root": "/" 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_count": 0, 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_pinned_count": 0, 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 14748, 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "inst": { 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "name": { 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "client", 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 14748 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-08-19T01:31:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935717940 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935717940 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "inst_str": "client.14748 192.168.0.1:0/1935717940", 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "addr_str": "192.168.0.1:0/1935717940", 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "inode_count": 1, 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_epoch": 17, 2024-08-19T01:31:15.149 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch": 89, 2024-08-19T01:31:15.150 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch_barrier": 0, 2024-08-19T01:31:15.150 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklisted": false, 2024-08-19T01:31:15.150 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs" 2024-08-19T01:31:15.150 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:31:15.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:15.156 DEBUG:teuthology.orchestra.run.smithi064:> stat --file-system '--printf=%T 2024-08-19T01:31:15.156 DEBUG:teuthology.orchestra.run.smithi064:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:15.171 INFO:teuthology.orchestra.run.smithi064.stdout:fuseblk 2024-08-19T01:31:15.171 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:15.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:15.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:15.236 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-08-19T01:31:15.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:15 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-19T01:31:15.341 INFO:teuthology.orchestra.run.smithi064.stdout:# file: . 2024-08-19T01:31:15.341 INFO:teuthology.orchestra.run.smithi064.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-08-19T01:31:15.341 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:31:15.347 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:31:15.347 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-19T01:31:15.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd blocklist ls 2024-08-19T01:31:15.664 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:15.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:15 smithi123 ceph-mon[21829]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:15 smithi161 ceph-mon[21757]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:15.955 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:15 smithi064 ceph-mon[18515]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 -- 172.21.15.64:0/2263214507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb301078b0 msgr2=0x7fbb30107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 --2- 172.21.15.64:0/2263214507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb301078b0 0x7fbb30107c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009930 tx=0x7fbb2002a420 comp rx=0 tx=0).stop 2024-08-19T01:31:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 -- 172.21.15.64:0/2263214507 shutdown_connections 2024-08-19T01:31:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 --2- 172.21.15.64:0/2263214507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb30102250 0x7fbb3010e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 --2- 172.21.15.64:0/2263214507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb301018b0 0x7fbb30101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 --2- 172.21.15.64:0/2263214507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb301078b0 0x7fbb30107c90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 -- 172.21.15.64:0/2263214507 >> 172.21.15.64:0/2263214507 conn(0x7fbb300fd5f0 msgr2=0x7fbb300ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 -- 172.21.15.64:0/2263214507 shutdown_connections 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.174+0000 7fbb34860640 1 -- 172.21.15.64:0/2263214507 wait complete. 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 Processor -- start 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 -- start start 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb301018b0 0x7fbb3019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb30102250 0x7fbb3019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb301078b0 0x7fbb3019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30111cb0 con 0x7fbb30102250 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30111b30 con 0x7fbb301018b0 2024-08-19T01:31:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.175+0000 7fbb34860640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30111e30 con 0x7fbb301078b0 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb30102250 0x7fbb3019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb30102250 0x7fbb3019a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54530/0 (socket says 172.21.15.64:54530) 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 -- 172.21.15.64:0/3106441916 learned_addr learned my addr 172.21.15.64:0/3106441916 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2e59c640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb301018b0 0x7fbb3019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2ed9d640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb301078b0 0x7fbb3019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 -- 172.21.15.64:0/3106441916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb301078b0 msgr2=0x7fbb3019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb301078b0 0x7fbb3019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 -- 172.21.15.64:0/3106441916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb301018b0 msgr2=0x7fbb3019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb301018b0 0x7fbb3019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 -- 172.21.15.64:0/3106441916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3019f3a0 con 0x7fbb30102250 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb2dd9b640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb30102250 0x7fbb3019a930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800d770 tx=0x7fbb1800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb277fe640 1 -- 172.21.15.64:0/3106441916 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18017070 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb277fe640 1 -- 172.21.15.64:0/3106441916 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb18005500 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3019f690 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb277fe640 1 -- 172.21.15.64:0/3106441916 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18004e00 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.176+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb301a6f70 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.177+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb30101d10 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.179+0000 7fbb277fe640 1 -- 172.21.15.64:0/3106441916 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbb18004fa0 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.180+0000 7fbb277fe640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fbb1c0a2b80 0x7fbb1c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.180+0000 7fbb2e59c640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fbb1c0a2b80 0x7fbb1c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.180+0000 7fbb277fe640 1 -- 172.21.15.64:0/3106441916 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbb180cf080 con 0x7fbb30102250 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.180+0000 7fbb2e59c640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fbb1c0a2b80 0x7fbb1c0a5040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009900 tx=0x7fbb200058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.183+0000 7fbb277fe640 1 -- 172.21.15.64:0/3106441916 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fbb1809b750 con 0x7fbb30102250 2024-08-19T01:31:16.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.372+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbb3019b750 con 0x7fbb30102250 2024-08-19T01:31:16.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.372+0000 7fbb277fe640 1 -- 172.21.15.64:0/3106441916 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7fbb3019b750 con 0x7fbb30102250 2024-08-19T01:31:16.374 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3202507182 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:16.374 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/2403632526 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:16.374 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1948699247 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1182511186 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/4090416923 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/973985936 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/3149805608 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/2403632526 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3838744191 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/3149805608 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:listed 10 entries 2024-08-19T01:31:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fbb1c0a2b80 msgr2=0x7fbb1c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fbb1c0a2b80 0x7fbb1c0a5040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009900 tx=0x7fbb200058d0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb30102250 msgr2=0x7fbb3019a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb30102250 0x7fbb3019a930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800d770 tx=0x7fbb1800dc30 comp rx=0 tx=0).stop 2024-08-19T01:31:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 shutdown_connections 2024-08-19T01:31:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fbb1c0a2b80 0x7fbb1c0a5040 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb301078b0 0x7fbb3019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb30102250 0x7fbb3019a930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 --2- 172.21.15.64:0/3106441916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb301018b0 0x7fbb3019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:16.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 >> 172.21.15.64:0/3106441916 conn(0x7fbb300fd5f0 msgr2=0x7fbb3010cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:16.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 shutdown_connections 2024-08-19T01:31:16.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:16.374+0000 7fbb34860640 1 -- 172.21.15.64:0/3106441916 wait complete. 2024-08-19T01:31:16.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:16 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3106441916' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:31:16.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:16 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3106441916' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:31:16.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:16 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3106441916' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:31:17.024 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:31:17.024 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-19T01:31:17.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd blocklist ls 2024-08-19T01:31:17.337 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:17.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:17 smithi064 ceph-mon[18515]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 -- 172.21.15.64:0/2666852141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c101150 msgr2=0x7f309c10d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 --2- 172.21.15.64:0/2666852141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c101150 0x7f309c10d680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3094009cd0 tx=0x7f309402a8e0 comp rx=0 tx=0).stop 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 -- 172.21.15.64:0/2666852141 shutdown_connections 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 --2- 172.21.15.64:0/2666852141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c101150 0x7f309c10d680 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 --2- 172.21.15.64:0/2666852141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f309c1007b0 0x7f309c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 --2- 172.21.15.64:0/2666852141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c1067b0 0x7f309c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 -- 172.21.15.64:0/2666852141 >> 172.21.15.64:0/2666852141 conn(0x7f309c0fc4d0 msgr2=0x7f309c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 -- 172.21.15.64:0/2666852141 shutdown_connections 2024-08-19T01:31:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 -- 172.21.15.64:0/2666852141 wait complete. 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.831+0000 7f30a47ea640 1 Processor -- start 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a47ea640 1 -- start start 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a47ea640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c1007b0 0x7f309c10e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a47ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c101150 0x7f309c10e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a47ea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f309c1067b0 0x7f309c10eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a47ea640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f309c111cd0 con 0x7f309c1007b0 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a47ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f309c111b50 con 0x7f309c101150 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a47ea640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f309c111e50 con 0x7f309c1067b0 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c1007b0 0x7f309c10e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c1007b0 0x7f309c10e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54542/0 (socket says 172.21.15.64:54542) 2024-08-19T01:31:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 -- 172.21.15.64:0/1579707318 learned_addr learned my addr 172.21.15.64:0/1579707318 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a1d85640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c101150 0x7f309c10e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2d87640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f309c1067b0 0x7f309c10eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 -- 172.21.15.64:0/1579707318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f309c1067b0 msgr2=0x7f309c10eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f309c1067b0 0x7f309c10eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 -- 172.21.15.64:0/1579707318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c101150 msgr2=0x7f309c10e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c101150 0x7f309c10e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.832+0000 7f30a2586640 1 -- 172.21.15.64:0/1579707318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309c1a65f0 con 0x7f309c1007b0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.833+0000 7f30a2586640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c1007b0 0x7f309c10e110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f309800d950 tx=0x7f309800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.833+0000 7f30937fe640 1 -- 172.21.15.64:0/1579707318 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098017070 con 0x7f309c1007b0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.833+0000 7f30937fe640 1 -- 172.21.15.64:0/1579707318 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3098009c30 con 0x7f309c1007b0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.833+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309c1a68e0 con 0x7f309c1007b0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.833+0000 7f30937fe640 1 -- 172.21.15.64:0/1579707318 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309800f6b0 con 0x7f309c1007b0 2024-08-19T01:31:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.833+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f309c1a6e20 con 0x7f309c1007b0 2024-08-19T01:31:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.834+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f309c100c10 con 0x7f309c1007b0 2024-08-19T01:31:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.835+0000 7f30937fe640 1 -- 172.21.15.64:0/1579707318 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f30980109d0 con 0x7f309c1007b0 2024-08-19T01:31:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.836+0000 7f30937fe640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f30880a2b80 0x7f30880a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.836+0000 7f30a1d85640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f30880a2b80 0x7f30880a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.836+0000 7f30937fe640 1 -- 172.21.15.64:0/1579707318 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f30980cee40 con 0x7f309c1007b0 2024-08-19T01:31:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.836+0000 7f30a1d85640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f30880a2b80 0x7f30880a5040 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f309c10fba0 tx=0x7f308c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:17.839+0000 7f30937fe640 1 -- 172.21.15.64:0/1579707318 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f3098009de0 con 0x7f309c1007b0 2024-08-19T01:31:17.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:17 smithi123 ceph-mon[21829]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:17 smithi161 ceph-mon[21757]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:18.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.029+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f309c1a7050 con 0x7f309c1007b0 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.029+0000 7f30937fe640 1 -- 172.21.15.64:0/1579707318 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7f309809bb60 con 0x7f309c1007b0 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3202507182 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/2403632526 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1948699247 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1182511186 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/4090416923 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/973985936 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/3149805608 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/2403632526 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3838744191 2024-08-20T01:24:33.597653+0000 2024-08-19T01:31:18.031 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/3149805608 2024-08-20T01:24:10.525240+0000 2024-08-19T01:31:18.032 INFO:teuthology.orchestra.run.smithi064.stderr:listed 10 entries 2024-08-19T01:31:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f30880a2b80 msgr2=0x7f30880a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f30880a2b80 0x7f30880a5040 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f309c10fba0 tx=0x7f308c00b390 comp rx=0 tx=0).stop 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c1007b0 msgr2=0x7f309c10e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c1007b0 0x7f309c10e110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f309800d950 tx=0x7f309800de10 comp rx=0 tx=0).stop 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 shutdown_connections 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f30880a2b80 0x7f30880a5040 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f309c1067b0 0x7f309c10eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c101150 0x7f309c10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 --2- 172.21.15.64:0/1579707318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f309c1007b0 0x7f309c10e110 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 >> 172.21.15.64:0/1579707318 conn(0x7f309c0fc4d0 msgr2=0x7f309c0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 shutdown_connections 2024-08-19T01:31:18.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:18.032+0000 7f30a47ea640 1 -- 172.21.15.64:0/1579707318 wait complete. 2024-08-19T01:31:18.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:18 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1579707318' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:31:18.825 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi064.front.sepia.ceph.com... 2024-08-19T01:31:18.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:18.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-08-19T01:31:18.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[59845]: fuse finished with error 0 and tester_r 0 2024-08-19T01:31:18.854 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T01:31:18.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:18 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1579707318' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:31:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:18 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1579707318' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:31:19.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:19 smithi123 ceph-mon[21829]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:19 smithi161 ceph-mon[21757]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:19.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:19 smithi064 ceph-mon[18515]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:31:21.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:21 smithi123 ceph-mon[21829]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-19T01:31:21.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:21 smithi161 ceph-mon[21757]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-19T01:31:21.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:21 smithi064 ceph-mon[18515]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-19T01:31:23.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:23 smithi123 ceph-mon[21829]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-19T01:31:23.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:23 smithi161 ceph-mon[21757]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-19T01:31:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:23 smithi064 ceph-mon[18515]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-08-19T01:31:24.955 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:31:24.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:31:24.955 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-08-19T01:31:24.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:24 smithi064 ceph-mon[18515]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:24.989 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:25.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:24 smithi123 ceph-mon[21829]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:25.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:24 smithi161 ceph-mon[21757]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:25.299 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 -- 172.21.15.64:0/3012931020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 msgr2=0x7f558010b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 --2- 172.21.15.64:0/3012931020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f558010b920 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f556c009900 tx=0x7f556c02a480 comp rx=0 tx=0).stop 2024-08-19T01:31:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 -- 172.21.15.64:0/3012931020 shutdown_connections 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 --2- 172.21.15.64:0/3012931020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f558010beb0 0x7f558010e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 --2- 172.21.15.64:0/3012931020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f558010b920 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 --2- 172.21.15.64:0/3012931020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5580102b10 0x7f5580102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 -- 172.21.15.64:0/3012931020 >> 172.21.15.64:0/3012931020 conn(0x7f55800fc4d0 msgr2=0x7f55800fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 -- 172.21.15.64:0/3012931020 shutdown_connections 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.776+0000 7f5586d48640 1 -- 172.21.15.64:0/3012931020 wait complete. 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 Processor -- start 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 -- start start 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5580102b10 0x7f5580100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f5580100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f558010beb0 0x7f55801011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5580111c60 con 0x7f5580103430 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5580111ae0 con 0x7f558010beb0 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5586d48640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5580111de0 con 0x7f5580102b10 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f5580100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f5580100cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42962/0 (socket says 172.21.15.64:42962) 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 -- 172.21.15.64:0/4041024480 learned_addr learned my addr 172.21.15.64:0/4041024480 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f55852e5640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f558010beb0 0x7f55801011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 -- 172.21.15.64:0/4041024480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5580102b10 msgr2=0x7f5580100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5584ae4640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5580102b10 0x7f5580100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5580102b10 0x7f5580100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 -- 172.21.15.64:0/4041024480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f558010beb0 msgr2=0x7f55801011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f558010beb0 0x7f55801011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 -- 172.21.15.64:0/4041024480 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55801a6550 con 0x7f5580103430 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f55852e5640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f558010beb0 0x7f55801011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f5584ae4640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5580102b10 0x7f5580100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.777+0000 7f557ffff640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f5580100cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f556c005ab0 tx=0x7f556c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.778+0000 7f557dffb640 1 -- 172.21.15.64:0/4041024480 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c03f030 con 0x7f5580103430 2024-08-19T01:31:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.778+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55801a67e0 con 0x7f5580103430 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.778+0000 7f557dffb640 1 -- 172.21.15.64:0/4041024480 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556c003ce0 con 0x7f5580103430 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.778+0000 7f557dffb640 1 -- 172.21.15.64:0/4041024480 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c035070 con 0x7f5580103430 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.778+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55801a6cc0 con 0x7f5580103430 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.779+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5580102ef0 con 0x7f5580103430 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.781+0000 7f557dffb640 1 -- 172.21.15.64:0/4041024480 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f556c042030 con 0x7f5580103430 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.781+0000 7f557dffb640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f55700a2b80 0x7f55700a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.781+0000 7f557dffb640 1 -- 172.21.15.64:0/4041024480 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f556c0efe70 con 0x7f5580103430 2024-08-19T01:31:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.781+0000 7f5584ae4640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f55700a2b80 0x7f55700a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.782+0000 7f5584ae4640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f55700a2b80 0x7f55700a5040 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f557400baa0 tx=0x7f5574005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.785+0000 7f557dffb640 1 -- 172.21.15.64:0/4041024480 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f556c0bcb90 con 0x7f5580103430 2024-08-19T01:31:25.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.962+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5580101ca0 con 0x7f5580103430 2024-08-19T01:31:25.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.964+0000 7f557dffb640 1 -- 172.21.15.64:0/4041024480 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7f556c0bc490 con 0x7f5580103430 2024-08-19T01:31:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:25.966 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":89,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:01.216103+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f55700a2b80 msgr2=0x7f55700a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f55700a2b80 0x7f55700a5040 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f557400baa0 tx=0x7f5574005d90 comp rx=0 tx=0).stop 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 msgr2=0x7f5580100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f5580100cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f556c005ab0 tx=0x7f556c0042f0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 shutdown_connections 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f55700a2b80 0x7f55700a5040 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f558010beb0 0x7f55801011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5580103430 0x7f5580100cb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 --2- 172.21.15.64:0/4041024480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5580102b10 0x7f5580100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 >> 172.21.15.64:0/4041024480 conn(0x7f55800fc4d0 msgr2=0x7f55800fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 shutdown_connections 2024-08-19T01:31:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:25.966+0000 7f5586d48640 1 -- 172.21.15.64:0/4041024480 wait complete. 2024-08-19T01:31:26.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:26.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:26 smithi064 ceph-mon[18515]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:26.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:26 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4041024480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:27.001 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:26 smithi161 ceph-mon[21757]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:27.001 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:26 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4041024480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:27.008 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:26 smithi123 ceph-mon[21829]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:26 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4041024480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 -- 172.21.15.64:0/2349859603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 msgr2=0x7fae98100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 --2- 172.21.15.64:0/2349859603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae98100f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fae88009900 tx=0x7fae8802a480 comp rx=0 tx=0).stop 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 -- 172.21.15.64:0/2349859603 shutdown_connections 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 --2- 172.21.15.64:0/2349859603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98102c20 0x7fae9810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 --2- 172.21.15.64:0/2349859603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae981022a0 0x7fae981026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 --2- 172.21.15.64:0/2349859603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae98100f30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 -- 172.21.15.64:0/2349859603 >> 172.21.15.64:0/2349859603 conn(0x7fae980fc500 msgr2=0x7fae980fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 -- 172.21.15.64:0/2349859603 shutdown_connections 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 -- 172.21.15.64:0/2349859603 wait complete. 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 Processor -- start 2024-08-19T01:31:27.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 -- start start 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.475+0000 7fae9e505640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae980fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae9e505640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae981022a0 0x7fae980fcef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae9e505640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98102c20 0x7fae980fd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae9e505640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fae98110cd0 con 0x7fae98100b50 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae9e505640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fae98110b50 con 0x7fae98102c20 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae9e505640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fae98110e50 con 0x7fae981022a0 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae980fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae980fc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42968/0 (socket says 172.21.15.64:42968) 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 -- 172.21.15.64:0/319267719 learned_addr learned my addr 172.21.15.64:0/319267719 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae8ffff640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae981022a0 0x7fae980fcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae9caa2640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98102c20 0x7fae980fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:27.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 -- 172.21.15.64:0/319267719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae981022a0 msgr2=0x7fae980fcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae981022a0 0x7fae980fcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 -- 172.21.15.64:0/319267719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98102c20 msgr2=0x7fae980fd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98102c20 0x7fae980fd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 -- 172.21.15.64:0/319267719 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae981a6730 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae97fff640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae980fc9b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fae88006010 tx=0x7fae88003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae95ffb640 1 -- 172.21.15.64:0/319267719 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8803f070 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae95ffb640 1 -- 172.21.15.64:0/319267719 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae88031e30 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae95ffb640 1 -- 172.21.15.64:0/319267719 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae88035030 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.476+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae981a69c0 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.477+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae981a6ef0 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.478+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae98100f30 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.479+0000 7fae95ffb640 1 -- 172.21.15.64:0/319267719 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fae88042070 con 0x7fae98100b50 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.479+0000 7fae95ffb640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fae840a2c00 0x7fae840a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:27.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.480+0000 7fae8ffff640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fae840a2c00 0x7fae840a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:27.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.480+0000 7fae95ffb640 1 -- 172.21.15.64:0/319267719 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fae880efaf0 con 0x7fae98100b50 2024-08-19T01:31:27.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.480+0000 7fae8ffff640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fae840a2c00 0x7fae840a50c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fae980fe440 tx=0x7fae80009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:27.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.484+0000 7fae95ffb640 1 -- 172.21.15.64:0/319267719 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fae880bc810 con 0x7fae98100b50 2024-08-19T01:31:27.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.660+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae98109dd0 con 0x7fae98100b50 2024-08-19T01:31:27.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.662+0000 7fae95ffb640 1 -- 172.21.15.64:0/319267719 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24117 (secure 0 0 0) 0x7fae880bc110 con 0x7fae98100b50 2024-08-19T01:31:27.663 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:27.664 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":89,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:01.216103+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fae840a2c00 msgr2=0x7fae840a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fae840a2c00 0x7fae840a50c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fae980fe440 tx=0x7fae80009110 comp rx=0 tx=0).stop 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 msgr2=0x7fae980fc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae980fc9b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fae88006010 tx=0x7fae88003c80 comp rx=0 tx=0).stop 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 shutdown_connections 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fae840a2c00 0x7fae840a50c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98102c20 0x7fae980fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae981022a0 0x7fae980fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 --2- 172.21.15.64:0/319267719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae98100b50 0x7fae980fc9b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 >> 172.21.15.64:0/319267719 conn(0x7fae980fc500 msgr2=0x7fae98069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:27.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 shutdown_connections 2024-08-19T01:31:27.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:27.664+0000 7fae9e505640 1 -- 172.21.15.64:0/319267719 wait complete. 2024-08-19T01:31:27.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:27 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/319267719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:27 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/319267719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:28.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:27 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/319267719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:28.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-08-19T01:31:28.567 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:28.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:28 smithi064 ceph-mon[18515]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/3419227771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a68102250 msgr2=0x7f6a6810e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/3419227771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a68102250 0x7f6a6810e780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009cd0 tx=0x7f6a6002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/3419227771 shutdown_connections 2024-08-19T01:31:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/3419227771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a68102250 0x7f6a6810e780 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/3419227771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a681018b0 0x7f6a68101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/3419227771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a681078b0 0x7f6a68107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/3419227771 >> 172.21.15.64:0/3419227771 conn(0x7f6a680fd5f0 msgr2=0x7f6a680ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/3419227771 shutdown_connections 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/3419227771 wait complete. 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 Processor -- start 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.019+0000 7f6a6f1f9640 1 -- start start 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6f1f9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a681018b0 0x7f6a68074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6f1f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a68102250 0x7f6a68074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6f1f9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a681078b0 0x7f6a68074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6f1f9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68111cb0 con 0x7f6a681018b0 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6f1f9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68111b30 con 0x7f6a681078b0 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6f1f9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68111e30 con 0x7f6a68102250 2024-08-19T01:31:29.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a681018b0 0x7f6a68074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a681018b0 0x7f6a68074220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50320/0 (socket says 172.21.15.64:50320) 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 -- 172.21.15.64:0/591466474 learned_addr learned my addr 172.21.15.64:0/591466474 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6d796640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a681078b0 0x7f6a68074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a67fff640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a68102250 0x7f6a68074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 -- 172.21.15.64:0/591466474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a68102250 msgr2=0x7f6a68074760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a68102250 0x7f6a68074760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 -- 172.21.15.64:0/591466474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a681078b0 msgr2=0x7f6a68074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a681078b0 0x7f6a68074ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 -- 172.21.15.64:0/591466474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a681a6630 con 0x7f6a681018b0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6cf95640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a681018b0 0x7f6a68074220 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00d950 tx=0x7f6a5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a65ffb640 1 -- 172.21.15.64:0/591466474 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c017070 con 0x7f6a681018b0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a65ffb640 1 -- 172.21.15.64:0/591466474 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a5c009c30 con 0x7f6a681018b0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a65ffb640 1 -- 172.21.15.64:0/591466474 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00f6b0 con 0x7f6a681018b0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.020+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a681a6920 con 0x7f6a681018b0 2024-08-19T01:31:29.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.021+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a681a6eb0 con 0x7f6a681018b0 2024-08-19T01:31:29.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.022+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a68101d10 con 0x7f6a681018b0 2024-08-19T01:31:29.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.023+0000 7f6a65ffb640 1 -- 172.21.15.64:0/591466474 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a5c0109d0 con 0x7f6a681018b0 2024-08-19T01:31:29.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.023+0000 7f6a65ffb640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a580a2c00 0x7f6a580a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:29.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.023+0000 7f6a67fff640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a580a2c00 0x7f6a580a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:29.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.023+0000 7f6a65ffb640 1 -- 172.21.15.64:0/591466474 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6a5c0cee00 con 0x7f6a681018b0 2024-08-19T01:31:29.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.024+0000 7f6a67fff640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a580a2c00 0x7f6a580a50c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6a54005fd0 tx=0x7f6a54005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:29.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.028+0000 7f6a65ffb640 1 -- 172.21.15.64:0/591466474 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6a5c009de0 con 0x7f6a681018b0 2024-08-19T01:31:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:28 smithi123 ceph-mon[21829]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:28 smithi161 ceph-mon[21757]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:29.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.241+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6a68109e50 con 0x7f6a681018b0 2024-08-19T01:31:29.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.722+0000 7f6a65ffb640 1 -- 172.21.15.64:0/591466474 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f6a5c09bac0 con 0x7f6a681018b0 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a580a2c00 msgr2=0x7f6a580a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a580a2c00 0x7f6a580a50c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6a54005fd0 tx=0x7f6a54005850 comp rx=0 tx=0).stop 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a681018b0 msgr2=0x7f6a68074220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a681018b0 0x7f6a68074220 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00d950 tx=0x7f6a5c00de10 comp rx=0 tx=0).stop 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 shutdown_connections 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a580a2c00 0x7f6a580a50c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a681078b0 0x7f6a68074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a68102250 0x7f6a68074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 --2- 172.21.15.64:0/591466474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a681018b0 0x7f6a68074220 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 >> 172.21.15.64:0/591466474 conn(0x7f6a680fd5f0 msgr2=0x7f6a680fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 shutdown_connections 2024-08-19T01:31:29.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:29.725+0000 7f6a6f1f9640 1 -- 172.21.15.64:0/591466474 wait complete. 2024-08-19T01:31:29.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:29 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/591466474' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-08-19T01:31:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:29 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/591466474' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-08-19T01:31:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:29 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/591466474' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-08-19T01:31:30.401 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-08-19T01:31:30.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-08-19T01:31:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:30.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:30 smithi064 ceph-mon[18515]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:30.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:30 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/591466474' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-08-19T01:31:30.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:30 smithi064 ceph-mon[18515]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:30 smithi123 ceph-mon[21829]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:31.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:30 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/591466474' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-08-19T01:31:31.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:30 smithi123 ceph-mon[21829]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 -- 172.21.15.64:0/1485346891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 msgr2=0x7feabc100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 --2- 172.21.15.64:0/1485346891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc100c10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feab0009930 tx=0x7feab002a420 comp rx=0 tx=0).stop 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 -- 172.21.15.64:0/1485346891 shutdown_connections 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 --2- 172.21.15.64:0/1485346891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feabc101150 0x7feabc10d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 --2- 172.21.15.64:0/1485346891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc100c10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 --2- 172.21.15.64:0/1485346891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feabc1067b0 0x7feabc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 -- 172.21.15.64:0/1485346891 >> 172.21.15.64:0/1485346891 conn(0x7feabc0fc4d0 msgr2=0x7feabc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 -- 172.21.15.64:0/1485346891 shutdown_connections 2024-08-19T01:31:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.161+0000 7feac1d3d640 1 -- 172.21.15.64:0/1485346891 wait complete. 2024-08-19T01:31:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 Processor -- start 2024-08-19T01:31:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 -- start start 2024-08-19T01:31:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feabc101150 0x7feabc197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feabc1067b0 0x7feabc19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feabc110c50 con 0x7feabc1007b0 2024-08-19T01:31:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feabc110ad0 con 0x7feabc101150 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feac1d3d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feabc110dd0 con 0x7feabc1067b0 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc1971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50330/0 (socket says 172.21.15.64:50330) 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabb7fe640 1 -- 172.21.15.64:0/3461344138 learned_addr learned my addr 172.21.15.64:0/3461344138 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabaffd640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feabc101150 0x7feabc197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabbfff640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feabc1067b0 0x7feabc19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabb7fe640 1 -- 172.21.15.64:0/3461344138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feabc1067b0 msgr2=0x7feabc19bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabb7fe640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feabc1067b0 0x7feabc19bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabb7fe640 1 -- 172.21.15.64:0/3461344138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feabc101150 msgr2=0x7feabc197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.162+0000 7feabb7fe640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feabc101150 0x7feabc197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feabb7fe640 1 -- 172.21.15.64:0/3461344138 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feabc19c1a0 con 0x7feabc1007b0 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feabbfff640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feabc1067b0 0x7feabc19bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feabb7fe640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc1971f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feaac00b4a0 tx=0x7feaac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feab8ff9640 1 -- 172.21.15.64:0/3461344138 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac014070 con 0x7feabc1007b0 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feab8ff9640 1 -- 172.21.15.64:0/3461344138 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac010af0 con 0x7feabc1007b0 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feabc19c490 con 0x7feabc1007b0 2024-08-19T01:31:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feab8ff9640 1 -- 172.21.15.64:0/3461344138 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac017d10 con 0x7feabc1007b0 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.163+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feabc1a3d70 con 0x7feabc1007b0 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.164+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feabc100c90 con 0x7feabc1007b0 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.165+0000 7feab8ff9640 1 -- 172.21.15.64:0/3461344138 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7feaac017070 con 0x7feabc1007b0 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.166+0000 7feab8ff9640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7feaa40a2b80 0x7feaa40a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.166+0000 7feabaffd640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7feaa40a2b80 0x7feaa40a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.166+0000 7feab8ff9640 1 -- 172.21.15.64:0/3461344138 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7feaac0ce650 con 0x7feabc1007b0 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.166+0000 7feabaffd640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7feaa40a2b80 0x7feaa40a5040 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feab0009900 tx=0x7feab00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.170+0000 7feab8ff9640 1 -- 172.21.15.64:0/3461344138 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7feaac09b370 con 0x7feabc1007b0 2024-08-19T01:31:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:30 smithi161 ceph-mon[21757]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-08-19T01:31:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:30 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/591466474' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-08-19T01:31:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:30 smithi161 ceph-mon[21757]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:31.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.384+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7feabc108da0 con 0x7feabc1007b0 2024-08-19T01:31:31.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.731+0000 7feab8ff9640 1 -- 172.21.15.64:0/3461344138 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7feaac09ac70 con 0x7feabc1007b0 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7feaa40a2b80 msgr2=0x7feaa40a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7feaa40a2b80 0x7feaa40a5040 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feab0009900 tx=0x7feab00058d0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 msgr2=0x7feabc1971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc1971f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feaac00b4a0 tx=0x7feaac00b960 comp rx=0 tx=0).stop 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 shutdown_connections 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7feaa40a2b80 0x7feaa40a5040 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feabc1067b0 0x7feabc19bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feabc101150 0x7feabc197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 --2- 172.21.15.64:0/3461344138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feabc1007b0 0x7feabc1971f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:31.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 >> 172.21.15.64:0/3461344138 conn(0x7feabc0fc4d0 msgr2=0x7feabc0fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:31.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 shutdown_connections 2024-08-19T01:31:31.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:31.733+0000 7feac1d3d640 1 -- 172.21.15.64:0/3461344138 wait complete. 2024-08-19T01:31:31.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:31 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3461344138' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-08-19T01:31:32.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:31 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3461344138' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-08-19T01:31:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:31 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3461344138' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-08-19T01:31:32.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-08-19T01:31:32.666 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:32.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:32 smithi064 ceph-mon[18515]: pgmap v288: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-08-19T01:31:32.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:32 smithi064 ceph-mon[18515]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-19T01:31:32.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:32 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3461344138' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-08-19T01:31:32.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:32 smithi064 ceph-mon[18515]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-08-19T01:31:32.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:32 smithi064 ceph-mon[18515]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:32.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:32 smithi064 ceph-mon[18515]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-08-19T01:31:32.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:32 smithi064 ceph-mon[18515]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-08-19T01:31:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:32 smithi123 ceph-mon[21829]: pgmap v288: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-08-19T01:31:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:32 smithi123 ceph-mon[21829]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-19T01:31:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:32 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3461344138' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-08-19T01:31:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:32 smithi123 ceph-mon[21829]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-08-19T01:31:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:32 smithi123 ceph-mon[21829]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:32 smithi123 ceph-mon[21829]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-08-19T01:31:33.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:32 smithi123 ceph-mon[21829]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-08-19T01:31:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:32 smithi161 ceph-mon[21757]: pgmap v288: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-08-19T01:31:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:32 smithi161 ceph-mon[21757]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-19T01:31:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:32 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3461344138' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-08-19T01:31:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:32 smithi161 ceph-mon[21757]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-08-19T01:31:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:32 smithi161 ceph-mon[21757]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-08-19T01:31:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:32 smithi161 ceph-mon[21757]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-08-19T01:31:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:32 smithi161 ceph-mon[21757]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-08-19T01:31:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 -- 172.21.15.64:0/2017344052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 msgr2=0x7f827410e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 --2- 172.21.15.64:0/2017344052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827410e340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f826c009ca0 tx=0x7f826c02a950 comp rx=0 tx=0).stop 2024-08-19T01:31:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 -- 172.21.15.64:0/2017344052 shutdown_connections 2024-08-19T01:31:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 --2- 172.21.15.64:0/2017344052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827410e340 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 --2- 172.21.15.64:0/2017344052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8274103410 0x7f827410b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 --2- 172.21.15.64:0/2017344052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8274102af0 0x7f8274102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 -- 172.21.15.64:0/2017344052 >> 172.21.15.64:0/2017344052 conn(0x7f82740fc4d0 msgr2=0x7f82740fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 -- 172.21.15.64:0/2017344052 shutdown_connections 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 -- 172.21.15.64:0/2017344052 wait complete. 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 Processor -- start 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.180+0000 7f8279812640 1 -- start start 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f8279812640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8274102af0 0x7f827419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f8279812640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8274103410 0x7f827419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f8279812640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f8279812640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8274111ce0 con 0x7f827410bed0 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f8279812640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8274111b60 con 0x7f8274103410 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f8279812640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8274111e60 con 0x7f8274102af0 2024-08-19T01:31:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827419ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50354/0 (socket says 172.21.15.64:50354) 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 -- 172.21.15.64:0/2266913877 learned_addr learned my addr 172.21.15.64:0/2266913877 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82727fc640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8274103410 0x7f827419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f8272ffd640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8274102af0 0x7f827419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 -- 172.21.15.64:0/2266913877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8274102af0 msgr2=0x7f827419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8274102af0 0x7f827419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 -- 172.21.15.64:0/2266913877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8274103410 msgr2=0x7f827419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8274103410 0x7f827419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 -- 172.21.15.64:0/2266913877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827419f330 con 0x7f827410bed0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.181+0000 7f82737fe640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827419ec50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f826c006010 tx=0x7f826c02f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.182+0000 7f8278810640 1 -- 172.21.15.64:0/2266913877 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c034040 con 0x7f827410bed0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.182+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827419f5c0 con 0x7f827410bed0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.182+0000 7f8278810640 1 -- 172.21.15.64:0/2266913877 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c003e00 con 0x7f827410bed0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.182+0000 7f8278810640 1 -- 172.21.15.64:0/2266913877 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c045480 con 0x7f827410bed0 2024-08-19T01:31:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.182+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82741a6f00 con 0x7f827410bed0 2024-08-19T01:31:33.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.183+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8274102ed0 con 0x7f827410bed0 2024-08-19T01:31:33.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.184+0000 7f8278810640 1 -- 172.21.15.64:0/2266913877 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f826c037020 con 0x7f827410bed0 2024-08-19T01:31:33.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.184+0000 7f8278810640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f825c0a2b80 0x7f825c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:33.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.184+0000 7f8272ffd640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f825c0a2b80 0x7f825c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:33.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.185+0000 7f8278810640 1 -- 172.21.15.64:0/2266913877 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f826c0f0350 con 0x7f827410bed0 2024-08-19T01:31:33.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.185+0000 7f8272ffd640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f825c0a2b80 0x7f825c0a5040 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8264009990 tx=0x7f8264009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:33.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.188+0000 7f8278810640 1 -- 172.21.15.64:0/2266913877 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f826c0bd070 con 0x7f827410bed0 2024-08-19T01:31:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.401+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f82741a70d0 con 0x7f827410bed0 2024-08-19T01:31:33.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.577+0000 7f8278810640 1 -- 172.21.15.64:0/2266913877 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f826c0bc970 con 0x7f827410bed0 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f825c0a2b80 msgr2=0x7f825c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f825c0a2b80 0x7f825c0a5040 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8264009990 tx=0x7f8264009110 comp rx=0 tx=0).stop 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 msgr2=0x7f827419ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827419ec50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f826c006010 tx=0x7f826c02f710 comp rx=0 tx=0).stop 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 shutdown_connections 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f825c0a2b80 0x7f825c0a5040 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827410bed0 0x7f827419ec50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8274103410 0x7f827419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 --2- 172.21.15.64:0/2266913877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8274102af0 0x7f827419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:33.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.579+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 >> 172.21.15.64:0/2266913877 conn(0x7f82740fc4d0 msgr2=0x7f82741017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:33.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.580+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 shutdown_connections 2024-08-19T01:31:33.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:33.580+0000 7f8279812640 1 -- 172.21.15.64:0/2266913877 wait complete. 2024-08-19T01:31:34.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] up:active 2024-08-19T01:31:34.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-08-19T01:31:34.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-19T01:31:34.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: Cluster is now healthy 2024-08-19T01:31:34.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-08-19T01:31:34.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-08-19T01:31:34.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-08-19T01:31:34.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2266913877' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-08-19T01:31:34.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2266913877' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-08-19T01:31:34.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] up:active 2024-08-19T01:31:34.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:31:34.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-08-19T01:31:34.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-08-19T01:31:34.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-08-19T01:31:34.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-08-19T01:31:34.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-08-19T01:31:34.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-08-19T01:31:34.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]: dispatch 2024-08-19T01:31:34.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-08-19T01:31:34.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-08-19T01:31:34.170 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:33 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-08-19T01:31:34.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] up:active 2024-08-19T01:31:34.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-08-19T01:31:34.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-19T01:31:34.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: Cluster is now healthy 2024-08-19T01:31:34.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-08-19T01:31:34.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-08-19T01:31:34.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-08-19T01:31:34.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2266913877' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-08-19T01:31:34.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2266913877' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] up:active 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-08-19T01:31:34.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-08-19T01:31:34.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]: dispatch 2024-08-19T01:31:34.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-08-19T01:31:34.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-08-19T01:31:34.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:33 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-08-19T01:31:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] up:active 2024-08-19T01:31:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-08-19T01:31:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-19T01:31:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: Cluster is now healthy 2024-08-19T01:31:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-08-19T01:31:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-08-19T01:31:34.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2266913877' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2266913877' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] up:active 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-08-19T01:31:34.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-08-19T01:31:34.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-08-19T01:31:34.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]: dispatch 2024-08-19T01:31:34.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-08-19T01:31:34.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-08-19T01:31:34.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:33 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-08-19T01:31:34.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-08-19T01:31:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:35.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 -- 172.21.15.64:0/2007261532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff774101150 msgr2=0x7ff77410d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/2007261532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff774101150 0x7ff77410d680 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009cd0 tx=0x7ff76c02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 -- 172.21.15.64:0/2007261532 shutdown_connections 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/2007261532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff774101150 0x7ff77410d680 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/2007261532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7741007b0 0x7ff774100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/2007261532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7741067b0 0x7ff774106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 -- 172.21.15.64:0/2007261532 >> 172.21.15.64:0/2007261532 conn(0x7ff7740fc4d0 msgr2=0x7ff7740fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.130+0000 7ff77c2b6640 1 -- 172.21.15.64:0/2007261532 shutdown_connections 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 -- 172.21.15.64:0/2007261532 wait complete. 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 Processor -- start 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 -- start start 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7741007b0 0x7ff774198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff774101150 0x7ff7741986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7741067b0 0x7ff77419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff774111cd0 con 0x7ff7741067b0 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff774111b50 con 0x7ff774101150 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77c2b6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff774111e50 con 0x7ff7741007b0 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77a853640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7741067b0 0x7ff77419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77a853640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7741067b0 0x7ff77419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50370/0 (socket says 172.21.15.64:50370) 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.131+0000 7ff77a853640 1 -- 172.21.15.64:0/178078084 learned_addr learned my addr 172.21.15.64:0/178078084 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77a052640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7741007b0 0x7ff774198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff779851640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff774101150 0x7ff7741986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77a853640 1 -- 172.21.15.64:0/178078084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7741007b0 msgr2=0x7ff774198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77a853640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7741007b0 0x7ff774198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77a853640 1 -- 172.21.15.64:0/178078084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff774101150 msgr2=0x7ff7741986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77a853640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff774101150 0x7ff7741986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77a853640 1 -- 172.21.15.64:0/178078084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77419d120 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff779851640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff774101150 0x7ff7741986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77a853640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7741067b0 0x7ff77419ca40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff76c005cc0 tx=0x7ff76c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff7677fe640 1 -- 172.21.15.64:0/178078084 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c004040 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff7677fe640 1 -- 172.21.15.64:0/178078084 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff76c0046c0 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77419d3b0 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff7677fe640 1 -- 172.21.15.64:0/178078084 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c046480 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.132+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7741a4cf0 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.133+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff774100c10 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.134+0000 7ff7677fe640 1 -- 172.21.15.64:0/178078084 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff76c0305a0 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.135+0000 7ff7677fe640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff7600a2c00 0x7ff7600a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.135+0000 7ff77a052640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff7600a2c00 0x7ff7600a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.135+0000 7ff7677fe640 1 -- 172.21.15.64:0/178078084 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7ff76c02e030 con 0x7ff7741067b0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.136+0000 7ff77a052640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff7600a2c00 0x7ff7600a50c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff770005fd0 tx=0x7ff770005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:35.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.139+0000 7ff7677fe640 1 -- 172.21.15.64:0/178078084 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff76c04b060 con 0x7ff7741067b0 2024-08-19T01:31:35.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.351+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7ff774108790 con 0x7ff7741067b0 2024-08-19T01:31:35.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.590+0000 7ff7677fe640 1 -- 172.21.15.64:0/178078084 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7ff76c0bd820 con 0x7ff7741067b0 2024-08-19T01:31:35.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.593+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff7600a2c00 msgr2=0x7ff7600a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:35.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.593+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff7600a2c00 0x7ff7600a50c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff770005fd0 tx=0x7ff770005850 comp rx=0 tx=0).stop 2024-08-19T01:31:35.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.593+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7741067b0 msgr2=0x7ff77419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:35.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.593+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7741067b0 0x7ff77419ca40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff76c005cc0 tx=0x7ff76c003970 comp rx=0 tx=0).stop 2024-08-19T01:31:35.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 shutdown_connections 2024-08-19T01:31:35.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff7600a2c00 0x7ff7600a50c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7741067b0 0x7ff77419ca40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff774101150 0x7ff7741986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 --2- 172.21.15.64:0/178078084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7741007b0 0x7ff774198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:35.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 >> 172.21.15.64:0/178078084 conn(0x7ff7740fc4d0 msgr2=0x7ff7740fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:35.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 shutdown_connections 2024-08-19T01:31:35.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:35.594+0000 7ff77c2b6640 1 -- 172.21.15.64:0/178078084 wait complete. 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: pgmap v289: 129 pgs: 129 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 2 op/s 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]': finished 2024-08-19T01:31:35.608 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-08-19T01:31:35.609 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-08-19T01:31:35.609 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-08-19T01:31:35.609 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: osdmap e90: 12 total, 12 up, 12 in 2024-08-19T01:31:35.609 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:35 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/178078084' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: pgmap v289: 129 pgs: 129 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 2 op/s 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-08-19T01:31:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]': finished 2024-08-19T01:31:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-08-19T01:31:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-08-19T01:31:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-08-19T01:31:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: osdmap e90: 12 total, 12 up, 12 in 2024-08-19T01:31:35.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:35 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/178078084' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: pgmap v289: 129 pgs: 129 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 2 op/s 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]': finished 2024-08-19T01:31:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-08-19T01:31:35.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-08-19T01:31:35.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-08-19T01:31:35.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: osdmap e90: 12 total, 12 up, 12 in 2024-08-19T01:31:35.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:35 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/178078084' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-08-19T01:31:36.294 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-08-19T01:31:36.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-08-19T01:31:36.555 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:36.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:36 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/178078084' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-08-19T01:31:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:36 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:31:36.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:36 smithi123 ceph-mon[21829]: osdmap e91: 12 total, 12 up, 12 in 2024-08-19T01:31:36.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:36 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/178078084' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-08-19T01:31:36.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:36 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:31:36.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:36 smithi064 ceph-mon[18515]: osdmap e91: 12 total, 12 up, 12 in 2024-08-19T01:31:36.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:36 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/178078084' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-08-19T01:31:36.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:36 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:31:36.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:36 smithi161 ceph-mon[21757]: osdmap e91: 12 total, 12 up, 12 in 2024-08-19T01:31:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 -- 172.21.15.64:0/2919341001 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0100b50 msgr2=0x7f42b0100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 --2- 172.21.15.64:0/2919341001 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0100b50 0x7f42b0100f30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009900 tx=0x7f42a002a480 comp rx=0 tx=0).stop 2024-08-19T01:31:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 -- 172.21.15.64:0/2919341001 shutdown_connections 2024-08-19T01:31:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 --2- 172.21.15.64:0/2919341001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42b0102c20 0x7f42b010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 --2- 172.21.15.64:0/2919341001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42b01022a0 0x7f42b01026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 --2- 172.21.15.64:0/2919341001 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0100b50 0x7f42b0100f30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 -- 172.21.15.64:0/2919341001 >> 172.21.15.64:0/2919341001 conn(0x7f42b00fc500 msgr2=0x7f42b00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 -- 172.21.15.64:0/2919341001 shutdown_connections 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.022+0000 7f42b68db640 1 -- 172.21.15.64:0/2919341001 wait complete. 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 Processor -- start 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 -- start start 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42b0100b50 0x7f42b01982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42b01022a0 0x7f42b0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0102c20 0x7f42b019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f42b0111dd0 con 0x7f42b0102c20 2024-08-19T01:31:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42b0111c50 con 0x7f42b0100b50 2024-08-19T01:31:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b68db640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42b0111f50 con 0x7f42b01022a0 2024-08-19T01:31:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b4e78640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0102c20 0x7f42b019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b4e78640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0102c20 0x7f42b019cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50396/0 (socket says 172.21.15.64:50396) 2024-08-19T01:31:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b4e78640 1 -- 172.21.15.64:0/2890764838 learned_addr learned my addr 172.21.15.64:0/2890764838 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42affff640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42b0100b50 0x7f42b01982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42af7fe640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42b01022a0 0x7f42b0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b4e78640 1 -- 172.21.15.64:0/2890764838 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42b01022a0 msgr2=0x7f42b0198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b4e78640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42b01022a0 0x7f42b0198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b4e78640 1 -- 172.21.15.64:0/2890764838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42b0100b50 msgr2=0x7f42b01982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.023+0000 7f42b4e78640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42b0100b50 0x7f42b01982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.024+0000 7f42b4e78640 1 -- 172.21.15.64:0/2890764838 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b019d2a0 con 0x7f42b0102c20 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.024+0000 7f42b4e78640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0102c20 0x7f42b019cbc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f42a800eaa0 tx=0x7f42a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.024+0000 7f42ad7fa640 1 -- 172.21.15.64:0/2890764838 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a800cdb0 con 0x7f42b0102c20 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.024+0000 7f42ad7fa640 1 -- 172.21.15.64:0/2890764838 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a8005020 con 0x7f42b0102c20 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.024+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b019d590 con 0x7f42b0102c20 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.024+0000 7f42ad7fa640 1 -- 172.21.15.64:0/2890764838 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a80116b0 con 0x7f42b0102c20 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.024+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b01a4e70 con 0x7f42b0102c20 2024-08-19T01:31:37.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.025+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42b0100f30 con 0x7f42b0102c20 2024-08-19T01:31:37.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.026+0000 7f42ad7fa640 1 -- 172.21.15.64:0/2890764838 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f42a8005560 con 0x7f42b0102c20 2024-08-19T01:31:37.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.027+0000 7f42ad7fa640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42980a2b80 0x7f42980a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:37.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.027+0000 7f42ad7fa640 1 -- 172.21.15.64:0/2890764838 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f42a80cf5f0 con 0x7f42b0102c20 2024-08-19T01:31:37.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.027+0000 7f42affff640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42980a2b80 0x7f42980a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:37.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.027+0000 7f42affff640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42980a2b80 0x7f42980a5040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009230 tx=0x7f42a00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:37.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.031+0000 7f42ad7fa640 1 -- 172.21.15.64:0/2890764838 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f42a8016030 con 0x7f42b0102c20 2024-08-19T01:31:37.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.230+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f42b0062360 con 0x7f42980a2b80 2024-08-19T01:31:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.257+0000 7f42ad7fa640 1 -- 172.21.15.64:0/2890764838 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f42b0062360 con 0x7f42980a2b80 2024-08-19T01:31:37.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42980a2b80 msgr2=0x7f42980a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:37.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42980a2b80 0x7f42980a5040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009230 tx=0x7f42a00058d0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0102c20 msgr2=0x7f42b019cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0102c20 0x7f42b019cbc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f42a800eaa0 tx=0x7f42a800ef60 comp rx=0 tx=0).stop 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 shutdown_connections 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f42980a2b80 0x7f42980a5040 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42b0102c20 0x7f42b019cbc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42b01022a0 0x7f42b0198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 --2- 172.21.15.64:0/2890764838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42b0100b50 0x7f42b01982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 >> 172.21.15.64:0/2890764838 conn(0x7f42b00fc500 msgr2=0x7f42b00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 shutdown_connections 2024-08-19T01:31:37.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:37.259+0000 7f42b68db640 1 -- 172.21.15.64:0/2890764838 wait complete. 2024-08-19T01:31:37.484 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:31:37 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:31:37.235+0000 7f70a78fe640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:31:37.485 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:31:37 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:31:37.235+0000 7f70a78fe640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:31:37.485 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:31:37 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:31:37.235+0000 7f70a78fe640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:31:37.485 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:31:37 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:31:37.235+0000 7f70a78fe640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:31:37.485 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:31:37 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:31:37.235+0000 7f70a78fe640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:31:37.600 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:31:37 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: 2024-08-19T01:31:37.347+0000 7ff772445640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-08-19T01:31:37.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:37 smithi123 ceph-mon[21829]: pgmap v292: 129 pgs: 2 peering, 127 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-08-19T01:31:37.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:37 smithi161 ceph-mon[21757]: pgmap v292: 129 pgs: 2 peering, 127 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-08-19T01:31:37.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:37 smithi064 ceph-mon[18515]: pgmap v292: 129 pgs: 2 peering, 127 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-08-19T01:31:38.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-08-19T01:31:38.289 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:38.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:38 smithi064 ceph-mon[18515]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.775+0000 7fe541005640 1 -- 172.21.15.64:0/2327658210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 msgr2=0x7fe53c10d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.775+0000 7fe541005640 1 --2- 172.21.15.64:0/2327658210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c10d680 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe534009930 tx=0x7fe53402a420 comp rx=0 tx=0).stop 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.775+0000 7fe541005640 1 -- 172.21.15.64:0/2327658210 shutdown_connections 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.775+0000 7fe541005640 1 --2- 172.21.15.64:0/2327658210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c10d680 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.775+0000 7fe541005640 1 --2- 172.21.15.64:0/2327658210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe53c1007b0 0x7fe53c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.775+0000 7fe541005640 1 --2- 172.21.15.64:0/2327658210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c1067b0 0x7fe53c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.775+0000 7fe541005640 1 -- 172.21.15.64:0/2327658210 >> 172.21.15.64:0/2327658210 conn(0x7fe53c0fc4d0 msgr2=0x7fe53c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.776+0000 7fe541005640 1 -- 172.21.15.64:0/2327658210 shutdown_connections 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.776+0000 7fe541005640 1 -- 172.21.15.64:0/2327658210 wait complete. 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.776+0000 7fe541005640 1 Processor -- start 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.776+0000 7fe541005640 1 -- start start 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe541005640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe53c1007b0 0x7fe53c197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe541005640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe541005640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c1067b0 0x7fe53c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe541005640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c110ca0 con 0x7fe53c101150 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe541005640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c110b20 con 0x7fe53c1007b0 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe541005640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c110e20 con 0x7fe53c1067b0 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c1976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37260/0 (socket says 172.21.15.64:37260) 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 -- 172.21.15.64:0/1252311812 learned_addr learned my addr 172.21.15.64:0/1252311812 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:38.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53ad9d640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe53c1007b0 0x7fe53c197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53b59e640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c1067b0 0x7fe53c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 -- 172.21.15.64:0/1252311812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c1067b0 msgr2=0x7fe53c19ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c1067b0 0x7fe53c19ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 -- 172.21.15.64:0/1252311812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe53c1007b0 msgr2=0x7fe53c197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe53c1007b0 0x7fe53c197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 -- 172.21.15.64:0/1252311812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53c19c140 con 0x7fe53c101150 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.777+0000 7fe53a59c640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c1976d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe52400b4a0 tx=0x7fe52400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.778+0000 7fe533fff640 1 -- 172.21.15.64:0/1252311812 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524014070 con 0x7fe53c101150 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.778+0000 7fe533fff640 1 -- 172.21.15.64:0/1252311812 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe524010af0 con 0x7fe53c101150 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.778+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53c19c430 con 0x7fe53c101150 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.778+0000 7fe533fff640 1 -- 172.21.15.64:0/1252311812 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524017d10 con 0x7fe53c101150 2024-08-19T01:31:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.778+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe53c1a3d10 con 0x7fe53c101150 2024-08-19T01:31:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.779+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe53c100c90 con 0x7fe53c101150 2024-08-19T01:31:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.780+0000 7fe533fff640 1 -- 172.21.15.64:0/1252311812 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe524017070 con 0x7fe53c101150 2024-08-19T01:31:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.781+0000 7fe533fff640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5280a2c10 0x7fe5280a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.781+0000 7fe53ad9d640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5280a2c10 0x7fe5280a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.781+0000 7fe533fff640 1 -- 172.21.15.64:0/1252311812 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe5240ce740 con 0x7fe53c101150 2024-08-19T01:31:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.781+0000 7fe53ad9d640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5280a2c10 0x7fe5280a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00a400 tx=0x7fe52c00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.785+0000 7fe533fff640 1 -- 172.21.15.64:0/1252311812 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe52409aca0 con 0x7fe53c101150 2024-08-19T01:31:38.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:38 smithi123 ceph-mon[21829]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:38.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:38 smithi161 ceph-mon[21757]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:38.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:38.986+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe53c062360 con 0x7fe5280a2c10 2024-08-19T01:31:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.011+0000 7fe533fff640 1 -- 172.21.15.64:0/1252311812 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe53c062360 con 0x7fe5280a2c10 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5280a2c10 msgr2=0x7fe5280a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5280a2c10 0x7fe5280a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00a400 tx=0x7fe52c00d040 comp rx=0 tx=0).stop 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 msgr2=0x7fe53c1976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c1976d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe52400b4a0 tx=0x7fe52400b960 comp rx=0 tx=0).stop 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 shutdown_connections 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5280a2c10 0x7fe5280a50d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c1067b0 0x7fe53c19ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe53c101150 0x7fe53c1976d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 --2- 172.21.15.64:0/1252311812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe53c1007b0 0x7fe53c197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 >> 172.21.15.64:0/1252311812 conn(0x7fe53c0fc4d0 msgr2=0x7fe53c0fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 shutdown_connections 2024-08-19T01:31:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:39.013+0000 7fe541005640 1 -- 172.21.15.64:0/1252311812 wait complete. 2024-08-19T01:31:39.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:39 smithi064 ceph-mon[18515]: pgmap v293: 129 pgs: 2 peering, 127 active+clean; 584 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.0 KiB/s wr, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-08-19T01:31:39.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:39 smithi064 ceph-mon[18515]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-08-19T01:31:39.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-08-19T01:31:39.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:39 smithi123 ceph-mon[21829]: pgmap v293: 129 pgs: 2 peering, 127 active+clean; 584 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.0 KiB/s wr, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-08-19T01:31:39.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:39 smithi123 ceph-mon[21829]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-08-19T01:31:39.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:39 smithi161 ceph-mon[21757]: pgmap v293: 129 pgs: 2 peering, 127 active+clean; 584 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3.0 KiB/s wr, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-08-19T01:31:39.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:39 smithi161 ceph-mon[21757]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-08-19T01:31:39.996 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:40.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.493+0000 7f95d9917640 1 -- 172.21.15.64:0/3538564956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4103410 msgr2=0x7f95d41078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:40.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.493+0000 7f95d9917640 1 --2- 172.21.15.64:0/3538564956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4103410 0x7f95d41078c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009930 tx=0x7f95bc02a420 comp rx=0 tx=0).stop 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 -- 172.21.15.64:0/3538564956 shutdown_connections 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 --2- 172.21.15.64:0/3538564956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4107fd0 0x7f95d410e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 --2- 172.21.15.64:0/3538564956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4103410 0x7f95d41078c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 --2- 172.21.15.64:0/3538564956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95d4102a60 0x7f95d4102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 -- 172.21.15.64:0/3538564956 >> 172.21.15.64:0/3538564956 conn(0x7f95d40fc500 msgr2=0x7f95d40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 -- 172.21.15.64:0/3538564956 shutdown_connections 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 -- 172.21.15.64:0/3538564956 wait complete. 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 Processor -- start 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 -- start start 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.494+0000 7f95d9917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95d4102a60 0x7f95d4198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d9917640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4103410 0x7f95d4198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d9917640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4107fd0 0x7f95d419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d9917640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f95d4111dd0 con 0x7f95d4107fd0 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d9917640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f95d4111c50 con 0x7f95d4103410 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d9917640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f95d4111f50 con 0x7f95d4102a60 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4107fd0 0x7f95d419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4107fd0 0x7f95d419cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37288/0 (socket says 172.21.15.64:37288) 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 -- 172.21.15.64:0/1281507483 learned_addr learned my addr 172.21.15.64:0/1281507483 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d2ffd640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95d4102a60 0x7f95d4198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d27fc640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4103410 0x7f95d4198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 -- 172.21.15.64:0/1281507483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95d4102a60 msgr2=0x7f95d4198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95d4102a60 0x7f95d4198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 -- 172.21.15.64:0/1281507483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4103410 msgr2=0x7f95d4198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4103410 0x7f95d4198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 -- 172.21.15.64:0/1281507483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d419d2e0 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d2ffd640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95d4102a60 0x7f95d4198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d37fe640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4107fd0 0x7f95d419cc00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009fd0 tx=0x7f95c400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d8915640 1 -- 172.21.15.64:0/1281507483 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c4017070 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d8915640 1 -- 172.21.15.64:0/1281507483 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95c4003e50 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d419d5d0 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d8915640 1 -- 172.21.15.64:0/1281507483 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c40106c0 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.495+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d41a4eb0 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.497+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95d406a580 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.498+0000 7f95d8915640 1 -- 172.21.15.64:0/1281507483 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f95c4003970 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.498+0000 7f95d8915640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f95c00a2c10 0x7f95c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.499+0000 7f95d8915640 1 -- 172.21.15.64:0/1281507483 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f95c40cf370 con 0x7f95d4107fd0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.499+0000 7f95d2ffd640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f95c00a2c10 0x7f95c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:40.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.499+0000 7f95d2ffd640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f95c00a2c10 0x7f95c00a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f95cc005fd0 tx=0x7f95cc005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:40.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.503+0000 7f95d8915640 1 -- 172.21.15.64:0/1281507483 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f95c409bdb0 con 0x7f95d4107fd0 2024-08-19T01:31:40.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.708+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f95d4062360 con 0x7f95c00a2c10 2024-08-19T01:31:40.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:40 smithi064 ceph-mon[18515]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:40.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.714+0000 7f95d8915640 1 -- 172.21.15.64:0/1281507483 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f95d4062360 con 0x7f95c00a2c10 2024-08-19T01:31:40.716 INFO:teuthology.orchestra.run.smithi064.stdout:/volumes/qa/sv_0/1d6602fc-2d40-4f8b-a3f7-b221c5c700df 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f95c00a2c10 msgr2=0x7f95c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f95c00a2c10 0x7f95c00a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f95cc005fd0 tx=0x7f95cc005ea0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4107fd0 msgr2=0x7f95d419cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4107fd0 0x7f95d419cc00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009fd0 tx=0x7f95c400b260 comp rx=0 tx=0).stop 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 shutdown_connections 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f95c00a2c10 0x7f95c00a50d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95d4107fd0 0x7f95d419cc00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4103410 0x7f95d4198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 --2- 172.21.15.64:0/1281507483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95d4102a60 0x7f95d4198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.716+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 >> 172.21.15.64:0/1281507483 conn(0x7f95d40fc500 msgr2=0x7f95d4105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:40.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.717+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 shutdown_connections 2024-08-19T01:31:40.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:40.717+0000 7f95d9917640 1 -- 172.21.15.64:0/1281507483 wait complete. 2024-08-19T01:31:40.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:40 smithi123 ceph-mon[21829]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:40.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:40 smithi161 ceph-mon[21757]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:41.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-08-19T01:31:41.693 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:41.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:41 smithi064 ceph-mon[18515]: pgmap v294: 129 pgs: 2 peering, 127 active+clean; 591 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.2 KiB/s wr, 14 op/s; 557 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:41.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:41 smithi123 ceph-mon[21829]: pgmap v294: 129 pgs: 2 peering, 127 active+clean; 591 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.2 KiB/s wr, 14 op/s; 557 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:41.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:41 smithi161 ceph-mon[21757]: pgmap v294: 129 pgs: 2 peering, 127 active+clean; 591 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.2 KiB/s wr, 14 op/s; 557 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 -- 172.21.15.64:0/881827891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb4102c20 msgr2=0x7f4fb410d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/881827891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb4102c20 0x7f4fb410d770 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009ca0 tx=0x7f4fac02a950 comp rx=0 tx=0).stop 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 -- 172.21.15.64:0/881827891 shutdown_connections 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/881827891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb4102c20 0x7f4fb410d770 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/881827891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fb41022a0 0x7f4fb41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/881827891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb4100b50 0x7f4fb4100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 -- 172.21.15.64:0/881827891 >> 172.21.15.64:0/881827891 conn(0x7f4fb40fc500 msgr2=0x7f4fb40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 -- 172.21.15.64:0/881827891 shutdown_connections 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 -- 172.21.15.64:0/881827891 wait complete. 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.151+0000 7f4fb98d2640 1 Processor -- start 2024-08-19T01:31:42.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb98d2640 1 -- start start 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb98d2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fb4100b50 0x7f4fb41991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb98d2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb41022a0 0x7f4fb41996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb98d2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb4102c20 0x7f4fb41a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb98d2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb4110d50 con 0x7f4fb41022a0 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb98d2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb4110bd0 con 0x7f4fb4102c20 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb98d2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb4110ed0 con 0x7f4fb4100b50 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb41022a0 0x7f4fb41996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb41022a0 0x7f4fb41996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37316/0 (socket says 172.21.15.64:37316) 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 -- 172.21.15.64:0/2980628068 learned_addr learned my addr 172.21.15.64:0/2980628068 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb37fe640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb4102c20 0x7f4fb41a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb2ffd640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fb4100b50 0x7f4fb41991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:42.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 -- 172.21.15.64:0/2980628068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fb4100b50 msgr2=0x7f4fb41991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fb4100b50 0x7f4fb41991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 -- 172.21.15.64:0/2980628068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb4102c20 msgr2=0x7f4fb41a4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb4102c20 0x7f4fb41a4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4faa7fc640 1 -- 172.21.15.64:0/2980628068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb41a4760 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.152+0000 7f4fb37fe640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb4102c20 0x7f4fb41a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.153+0000 7f4faa7fc640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb41022a0 0x7f4fb41996e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00b4a0 tx=0x7f4f9c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.153+0000 7f4fb88d0640 1 -- 172.21.15.64:0/2980628068 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c014070 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.153+0000 7f4fb88d0640 1 -- 172.21.15.64:0/2980628068 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f9c010af0 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.153+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb41a4930 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.153+0000 7f4fb88d0640 1 -- 172.21.15.64:0/2980628068 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c017d10 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.153+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb41a4e70 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.154+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb4100fb0 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.155+0000 7f4fb88d0640 1 -- 172.21.15.64:0/2980628068 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4f9c017070 con 0x7f4fb41022a0 2024-08-19T01:31:42.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.156+0000 7f4fb88d0640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f4fa00a2c10 0x7f4fa00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:42.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.156+0000 7f4fb2ffd640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f4fa00a2c10 0x7f4fa00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:42.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.156+0000 7f4fb88d0640 1 -- 172.21.15.64:0/2980628068 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4f9c0ce8a0 con 0x7f4fb41022a0 2024-08-19T01:31:42.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.156+0000 7f4fb2ffd640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f4fa00a2c10 0x7f4fa00a50d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4009890 tx=0x7f4fa4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:42.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.160+0000 7f4fb88d0640 1 -- 172.21.15.64:0/2980628068 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f4f9c09b3f0 con 0x7f4fb41022a0 2024-08-19T01:31:42.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.360+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4fb4108e30 con 0x7f4fa00a2c10 2024-08-19T01:31:42.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.386+0000 7f4fb88d0640 1 -- 172.21.15.64:0/2980628068 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4fa4009480 con 0x7f4fa00a2c10 2024-08-19T01:31:42.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f4fa00a2c10 msgr2=0x7f4fa00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:42.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f4fa00a2c10 0x7f4fa00a50d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4009890 tx=0x7f4fa4008040 comp rx=0 tx=0).stop 2024-08-19T01:31:42.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb41022a0 msgr2=0x7f4fb41996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:42.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb41022a0 0x7f4fb41996e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00b4a0 tx=0x7f4f9c00b960 comp rx=0 tx=0).stop 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 shutdown_connections 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f4fa00a2c10 0x7f4fa00a50d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb4102c20 0x7f4fb41a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fb41022a0 0x7f4fb41996e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 --2- 172.21.15.64:0/2980628068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fb4100b50 0x7f4fb41991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 >> 172.21.15.64:0/2980628068 conn(0x7f4fb40fc500 msgr2=0x7f4fb40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 shutdown_connections 2024-08-19T01:31:42.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:42.388+0000 7f4fb98d2640 1 -- 172.21.15.64:0/2980628068 wait complete. 2024-08-19T01:31:42.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:42 smithi123 ceph-mon[21829]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:42.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:42 smithi161 ceph-mon[21757]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:42 smithi064 ceph-mon[18515]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:43.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-08-19T01:31:43.428 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:43.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:43 smithi064 ceph-mon[18515]: pgmap v295: 129 pgs: 129 active+clean; 591 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.0 KiB/s wr, 13 op/s; 557 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:43.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:43 smithi064 ceph-mon[18515]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:43.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:43 smithi123 ceph-mon[21829]: pgmap v295: 129 pgs: 129 active+clean; 591 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.0 KiB/s wr, 13 op/s; 557 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:43.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:43 smithi123 ceph-mon[21829]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:43 smithi161 ceph-mon[21757]: pgmap v295: 129 pgs: 129 active+clean; 591 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.0 KiB/s wr, 13 op/s; 557 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:43 smithi161 ceph-mon[21757]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:43.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 -- 172.21.15.64:0/960379977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c30102b00 msgr2=0x7f1c30102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:43.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 --2- 172.21.15.64:0/960379977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c30102b00 0x7f1c30102ee0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c009930 tx=0x7f1c2c02a420 comp rx=0 tx=0).stop 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 -- 172.21.15.64:0/960379977 shutdown_connections 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 --2- 172.21.15.64:0/960379977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c30108030 0x7f1c3010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 --2- 172.21.15.64:0/960379977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c301034b0 0x7f1c301078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 --2- 172.21.15.64:0/960379977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c30102b00 0x7f1c30102ee0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 -- 172.21.15.64:0/960379977 >> 172.21.15.64:0/960379977 conn(0x7f1c300fc500 msgr2=0x7f1c300fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 -- 172.21.15.64:0/960379977 shutdown_connections 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.948+0000 7f1c379e6640 1 -- 172.21.15.64:0/960379977 wait complete. 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 Processor -- start 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 -- start start 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c30102b00 0x7f1c30198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301034b0 0x7f1c30198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c30108030 0x7f1c3019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:43.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1c30111e00 con 0x7f1c301034b0 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c30111c80 con 0x7f1c30108030 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c379e6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c30111f80 con 0x7f1c30102b00 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c34f81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301034b0 0x7f1c30198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c34f81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301034b0 0x7f1c30198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37324/0 (socket says 172.21.15.64:37324) 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.949+0000 7f1c34f81640 1 -- 172.21.15.64:0/2470241210 learned_addr learned my addr 172.21.15.64:0/2470241210 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c35782640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c30102b00 0x7f1c30198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c35f83640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c30108030 0x7f1c3019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c34f81640 1 -- 172.21.15.64:0/2470241210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c30102b00 msgr2=0x7f1c30198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c34f81640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c30102b00 0x7f1c30198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c34f81640 1 -- 172.21.15.64:0/2470241210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c30108030 msgr2=0x7f1c3019cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c34f81640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c30108030 0x7f1c3019cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c34f81640 1 -- 172.21.15.64:0/2470241210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3019d2b0 con 0x7f1c301034b0 2024-08-19T01:31:43.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c35f83640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c30108030 0x7f1c3019cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c34f81640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301034b0 0x7f1c30198840 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000c970 tx=0x7f1c2000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c267fc640 1 -- 172.21.15.64:0/2470241210 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20007bb0 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3019d5a0 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c267fc640 1 -- 172.21.15.64:0/2470241210 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c20007d50 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c267fc640 1 -- 172.21.15.64:0/2470241210 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000f450 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.950+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c301a4e80 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.951+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c3006a4e0 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.952+0000 7f1c267fc640 1 -- 172.21.15.64:0/2470241210 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1c20004f60 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.953+0000 7f1c267fc640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1c1c0a2c10 0x7f1c1c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.953+0000 7f1c35782640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1c1c0a2c10 0x7f1c1c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.953+0000 7f1c267fc640 1 -- 172.21.15.64:0/2470241210 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1c200cf2f0 con 0x7f1c301034b0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.953+0000 7f1c35782640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1c1c0a2c10 0x7f1c1c0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c0096c0 tx=0x7f1c2c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:43.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:43.957+0000 7f1c267fc640 1 -- 172.21.15.64:0/2470241210 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1c20014030 con 0x7f1c301034b0 2024-08-19T01:31:44.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.164+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f1c30062360 con 0x7f1c1c0a2c10 2024-08-19T01:31:44.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.168+0000 7f1c267fc640 1 -- 172.21.15.64:0/2470241210 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1c30062360 con 0x7f1c1c0a2c10 2024-08-19T01:31:44.169 INFO:teuthology.orchestra.run.smithi064.stdout:/volumes/qa/sv_1/9de8b1b3-1cf7-4bdf-a4f3-7b1fad68ee8a 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1c1c0a2c10 msgr2=0x7f1c1c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1c1c0a2c10 0x7f1c1c0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c0096c0 tx=0x7f1c2c000bc0 comp rx=0 tx=0).stop 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301034b0 msgr2=0x7f1c30198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301034b0 0x7f1c30198840 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000c970 tx=0x7f1c2000ce30 comp rx=0 tx=0).stop 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 shutdown_connections 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1c1c0a2c10 0x7f1c1c0a50d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c30108030 0x7f1c3019cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301034b0 0x7f1c30198840 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 --2- 172.21.15.64:0/2470241210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c30102b00 0x7f1c30198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 >> 172.21.15.64:0/2470241210 conn(0x7f1c300fc500 msgr2=0x7f1c30105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:44.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 shutdown_connections 2024-08-19T01:31:44.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:44.170+0000 7f1c379e6640 1 -- 172.21.15.64:0/2470241210 wait complete. 2024-08-19T01:31:44.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:31:45.168 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:45.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 -- 172.21.15.64:0/3711139549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400102a70 msgr2=0x7fe400102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3711139549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400102a70 0x7fe400102e50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009ca0 tx=0x7fe3f802a950 comp rx=0 tx=0).stop 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 -- 172.21.15.64:0/3711139549 shutdown_connections 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3711139549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe40010bea0 0x7fe40010e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3711139549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400103390 0x7fe40010b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3711139549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400102a70 0x7fe400102e50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 -- 172.21.15.64:0/3711139549 >> 172.21.15.64:0/3711139549 conn(0x7fe4000fc4d0 msgr2=0x7fe4000fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 -- 172.21.15.64:0/3711139549 shutdown_connections 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.627+0000 7fe406bbf640 1 -- 172.21.15.64:0/3711139549 wait complete. 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 Processor -- start 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 -- start start 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400102a70 0x7fe400198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400103390 0x7fe4001986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:45.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe40010bea0 0x7fe40019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe400111ca0 con 0x7fe400103390 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe400111b20 con 0x7fe400102a70 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe406bbf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe400111e20 con 0x7fe40010bea0 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400103390 0x7fe4001986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400103390 0x7fe4001986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37344/0 (socket says 172.21.15.64:37344) 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 -- 172.21.15.64:0/3362124963 learned_addr learned my addr 172.21.15.64:0/3362124963 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe40515c640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe40010bea0 0x7fe40019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe40495b640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400102a70 0x7fe400198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 -- 172.21.15.64:0/3362124963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe40010bea0 msgr2=0x7fe40019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe40010bea0 0x7fe40019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 -- 172.21.15.64:0/3362124963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400102a70 msgr2=0x7fe400198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400102a70 0x7fe400198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.628+0000 7fe3fffff640 1 -- 172.21.15.64:0/3362124963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40019d130 con 0x7fe400103390 2024-08-19T01:31:45.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.629+0000 7fe3fffff640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400103390 0x7fe4001986c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c940 tx=0x7fe3f400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.629+0000 7fe3fdffb640 1 -- 172.21.15.64:0/3362124963 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4004220 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.629+0000 7fe3fdffb640 1 -- 172.21.15.64:0/3362124963 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f40043c0 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.629+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40019d420 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.629+0000 7fe3fdffb640 1 -- 172.21.15.64:0/3362124963 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4004a00 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.629+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4001a4d00 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.630+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe400102e50 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.631+0000 7fe3fdffb640 1 -- 172.21.15.64:0/3362124963 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe3f40079b0 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.632+0000 7fe3fdffb640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe3e80a2c10 0x7fe3e80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.632+0000 7fe40495b640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe3e80a2c10 0x7fe3e80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.632+0000 7fe3fdffb640 1 -- 172.21.15.64:0/3362124963 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe3f40cf480 con 0x7fe400103390 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.632+0000 7fe40495b640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe3e80a2c10 0x7fe3e80a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8004ad0 tx=0x7fe3f8004a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:45.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.636+0000 7fe3fdffb640 1 -- 172.21.15.64:0/3362124963 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe3f4014030 con 0x7fe400103390 2024-08-19T01:31:45.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.849+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4001069b0 con 0x7fe400103390 2024-08-19T01:31:45.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:45 smithi064 ceph-mon[18515]: pgmap v296: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.5 KiB/s wr, 15 op/s; 490 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:45.850 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:45 smithi064 ceph-mon[18515]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:45.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.849+0000 7fe3fdffb640 1 -- 172.21.15.64:0/3362124963 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11921 (secure 0 0 0) 0x7fe3f409b8d0 con 0x7fe400103390 2024-08-19T01:31:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:45.851 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"btime":"2024-08-19T01:31:35:584965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:35.353608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:31:45.852 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 24 2024-08-19T01:31:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe3e80a2c10 msgr2=0x7fe3e80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe3e80a2c10 0x7fe3e80a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8004ad0 tx=0x7fe3f8004a60 comp rx=0 tx=0).stop 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400103390 msgr2=0x7fe4001986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400103390 0x7fe4001986c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c940 tx=0x7fe3f400ce00 comp rx=0 tx=0).stop 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 shutdown_connections 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe3e80a2c10 0x7fe3e80a50d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe40010bea0 0x7fe40019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe400103390 0x7fe4001986c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 --2- 172.21.15.64:0/3362124963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400102a70 0x7fe400198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 >> 172.21.15.64:0/3362124963 conn(0x7fe4000fc4d0 msgr2=0x7fe4000fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 shutdown_connections 2024-08-19T01:31:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:45.852+0000 7fe406bbf640 1 -- 172.21.15.64:0/3362124963 wait complete. 2024-08-19T01:31:45.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:45 smithi123 ceph-mon[21829]: pgmap v296: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.5 KiB/s wr, 15 op/s; 490 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:45.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:45 smithi123 ceph-mon[21829]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:45.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:45 smithi161 ceph-mon[21757]: pgmap v296: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.5 KiB/s wr, 15 op/s; 490 B/s, 1 keys/s, 1 objects/s recovering 2024-08-19T01:31:45.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:45 smithi161 ceph-mon[21757]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:31:46.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:46.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:46 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3362124963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:46.933 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:46.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:46 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3362124963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:46.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:46 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3362124963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 -- 172.21.15.64:0/1039779992 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c103390 msgr2=0x7f524c10b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 --2- 172.21.15.64:0/1039779992 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c103390 0x7f524c10b880 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5240009900 tx=0x7f524002a480 comp rx=0 tx=0).stop 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 -- 172.21.15.64:0/1039779992 shutdown_connections 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 --2- 172.21.15.64:0/1039779992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f524c10bea0 0x7f524c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 --2- 172.21.15.64:0/1039779992 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c103390 0x7f524c10b880 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 --2- 172.21.15.64:0/1039779992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524c102a70 0x7f524c102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 -- 172.21.15.64:0/1039779992 >> 172.21.15.64:0/1039779992 conn(0x7f524c0fc4d0 msgr2=0x7f524c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 -- 172.21.15.64:0/1039779992 shutdown_connections 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.418+0000 7f5252fe3640 1 -- 172.21.15.64:0/1039779992 wait complete. 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 Processor -- start 2024-08-19T01:31:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 -- start start 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c102a70 0x7f524c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524c103390 0x7f524c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f524c10bea0 0x7f524c19ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f524c111ca0 con 0x7f524c102a70 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f524c111b20 con 0x7f524c103390 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5252fe3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f524c111e20 con 0x7f524c10bea0 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5250d7f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c102a70 0x7f524c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5250d7f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c102a70 0x7f524c1981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37356/0 (socket says 172.21.15.64:37356) 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.419+0000 7f5250d7f640 1 -- 172.21.15.64:0/143153782 learned_addr learned my addr 172.21.15.64:0/143153782 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5250d7f640 1 -- 172.21.15.64:0/143153782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f524c10bea0 msgr2=0x7f524c19ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f524bfff640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524c103390 0x7f524c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:47.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5250d7f640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f524c10bea0 0x7f524c19ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5250d7f640 1 -- 172.21.15.64:0/143153782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524c103390 msgr2=0x7f524c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5250d7f640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524c103390 0x7f524c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5250d7f640 1 -- 172.21.15.64:0/143153782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524c19d150 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5250d7f640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c102a70 0x7f524c1981a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f523c00d950 tx=0x7f523c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5249ffb640 1 -- 172.21.15.64:0/143153782 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c017070 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5249ffb640 1 -- 172.21.15.64:0/143153782 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523c009c30 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524c19d440 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5249ffb640 1 -- 172.21.15.64:0/143153782 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c00f6b0 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.420+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f524c1a4d20 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.421+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f524c102e50 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.422+0000 7f5249ffb640 1 -- 172.21.15.64:0/143153782 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f523c0109d0 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.423+0000 7f5249ffb640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f52340a2c90 0x7f52340a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.423+0000 7f524bfff640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f52340a2c90 0x7f52340a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.423+0000 7f5249ffb640 1 -- 172.21.15.64:0/143153782 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f523c0cf230 con 0x7f524c102a70 2024-08-19T01:31:47.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.423+0000 7f524bfff640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f52340a2c90 0x7f52340a5150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52400096c0 tx=0x7f5240000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:47.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.427+0000 7f5249ffb640 1 -- 172.21.15.64:0/143153782 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f523c009de0 con 0x7f524c102a70 2024-08-19T01:31:47.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.607+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f524c02d5e0 con 0x7f524c102a70 2024-08-19T01:31:47.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.608+0000 7f5249ffb640 1 -- 172.21.15.64:0/143153782 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24643 (secure 0 0 0) 0x7f523c0bb070 con 0x7f524c102a70 2024-08-19T01:31:47.610 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:47.614 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":91,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:35.585064+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f52340a2c90 msgr2=0x7f52340a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f52340a2c90 0x7f52340a5150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52400096c0 tx=0x7f5240000bc0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c102a70 msgr2=0x7f524c1981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c102a70 0x7f524c1981a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f523c00d950 tx=0x7f523c00de10 comp rx=0 tx=0).stop 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 shutdown_connections 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f52340a2c90 0x7f52340a5150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f524c10bea0 0x7f524c19ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524c103390 0x7f524c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 --2- 172.21.15.64:0/143153782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f524c102a70 0x7f524c1981a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.611+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 >> 172.21.15.64:0/143153782 conn(0x7f524c0fc4d0 msgr2=0x7f524c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.612+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 shutdown_connections 2024-08-19T01:31:47.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:47.612+0000 7f5252fe3640 1 -- 172.21.15.64:0/143153782 wait complete. 2024-08-19T01:31:47.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:47 smithi123 ceph-mon[21829]: pgmap v297: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.0 KiB/s wr, 15 op/s; 442 B/s, 0 keys/s, 1 objects/s recovering 2024-08-19T01:31:47.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:47 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/143153782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:47.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:47 smithi161 ceph-mon[21757]: pgmap v297: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.0 KiB/s wr, 15 op/s; 442 B/s, 0 keys/s, 1 objects/s recovering 2024-08-19T01:31:47.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:47 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/143153782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:47 smithi064 ceph-mon[18515]: pgmap v297: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.0 KiB/s wr, 15 op/s; 442 B/s, 0 keys/s, 1 objects/s recovering 2024-08-19T01:31:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:47 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/143153782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:48.354 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-19T01:31:48.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T01:31:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:48 smithi123 ceph-mon[21829]: pgmap v298: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.2 KiB/s wr, 9 op/s; 371 B/s, 0 keys/s, 0 objects/s recovering 2024-08-19T01:31:48.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:48 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:31:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:48 smithi161 ceph-mon[21757]: pgmap v298: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.2 KiB/s wr, 9 op/s; 371 B/s, 0 keys/s, 0 objects/s recovering 2024-08-19T01:31:48.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:48 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:31:48.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:48 smithi064 ceph-mon[18515]: pgmap v298: 129 pgs: 129 active+clean; 623 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.2 KiB/s wr, 9 op/s; 371 B/s, 0 keys/s, 0 objects/s recovering 2024-08-19T01:31:48.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:48 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:31:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 -- 172.21.15.64:0/3441936095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce010beb0 msgr2=0x7f3ce010e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 --2- 172.21.15.64:0/3441936095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce010beb0 0x7f3ce010e330 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009ca0 tx=0x7f3cd802a950 comp rx=0 tx=0).stop 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 -- 172.21.15.64:0/3441936095 shutdown_connections 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 --2- 172.21.15.64:0/3441936095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce010beb0 0x7f3ce010e330 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 --2- 172.21.15.64:0/3441936095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce01033f0 0x7f3ce010b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 --2- 172.21.15.64:0/3441936095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce0102ad0 0x7f3ce0102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 -- 172.21.15.64:0/3441936095 >> 172.21.15.64:0/3441936095 conn(0x7f3ce00fc4d0 msgr2=0x7f3ce00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 -- 172.21.15.64:0/3441936095 shutdown_connections 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.778+0000 7f3ce6214640 1 -- 172.21.15.64:0/3441936095 wait complete. 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 Processor -- start 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 -- start start 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce0102ad0 0x7f3ce0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce01033f0 0x7f3ce0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce010beb0 0x7f3ce019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce0111c50 con 0x7f3ce0102ad0 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce0111ad0 con 0x7f3ce010beb0 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.779+0000 7f3ce6214640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce0111dd0 con 0x7f3ce01033f0 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce0102ad0 0x7f3ce0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce0102ad0 0x7f3ce0198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38396/0 (socket says 172.21.15.64:38396) 2024-08-19T01:31:49.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 -- 172.21.15.64:0/1081381642 learned_addr learned my addr 172.21.15.64:0/1081381642 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdeffd640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce01033f0 0x7f3ce0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdffff640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce010beb0 0x7f3ce019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 -- 172.21.15.64:0/1081381642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce01033f0 msgr2=0x7f3ce0198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce01033f0 0x7f3ce0198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 -- 172.21.15.64:0/1081381642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce010beb0 msgr2=0x7f3ce019c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce010beb0 0x7f3ce019c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 -- 172.21.15.64:0/1081381642 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce019d0c0 con 0x7f3ce0102ad0 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdf7fe640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce0102ad0 0x7f3ce0198110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000d950 tx=0x7f3cd000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdcff9640 1 -- 172.21.15.64:0/1081381642 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0017030 con 0x7f3ce0102ad0 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdcff9640 1 -- 172.21.15.64:0/1081381642 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cd0004020 con 0x7f3ce0102ad0 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce019d3b0 con 0x7f3ce0102ad0 2024-08-19T01:31:49.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3cdcff9640 1 -- 172.21.15.64:0/1081381642 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd00047a0 con 0x7f3ce0102ad0 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.780+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce01a4c90 con 0x7f3ce0102ad0 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.781+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce0102eb0 con 0x7f3ce0102ad0 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.782+0000 7f3cdcff9640 1 -- 172.21.15.64:0/1081381642 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3cd00041d0 con 0x7f3ce0102ad0 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.783+0000 7f3cdcff9640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3ccc0a2c10 0x7f3ccc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.783+0000 7f3cdeffd640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3ccc0a2c10 0x7f3ccc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.783+0000 7f3cdcff9640 1 -- 172.21.15.64:0/1081381642 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3cd00cee60 con 0x7f3ce0102ad0 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.784+0000 7f3cdeffd640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3ccc0a2c10 0x7f3ccc0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00fea60 tx=0x7f3cc8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:49.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.787+0000 7f3cdcff9640 1 -- 172.21.15.64:0/1081381642 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f3cd009b900 con 0x7f3ce0102ad0 2024-08-19T01:31:49.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.972+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ce0062360 con 0x7f3ccc0a2c10 2024-08-19T01:31:49.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.982+0000 7f3cdcff9640 1 -- 172.21.15.64:0/1081381642 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340639 (secure 0 0 0) 0x7f3ce0062360 con 0x7f3ccc0a2c10 2024-08-19T01:31:49.987 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:49.989 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-08-19T01:31:49.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3ccc0a2c10 msgr2=0x7f3ccc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:49.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3ccc0a2c10 0x7f3ccc0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00fea60 tx=0x7f3cc8005f90 comp rx=0 tx=0).stop 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce0102ad0 msgr2=0x7f3ce0198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce0102ad0 0x7f3ce0198110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000d950 tx=0x7f3cd000de10 comp rx=0 tx=0).stop 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 shutdown_connections 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3ccc0a2c10 0x7f3ccc0a50d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce010beb0 0x7f3ce019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce01033f0 0x7f3ce0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 --2- 172.21.15.64:0/1081381642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ce0102ad0 0x7f3ce0198110 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 >> 172.21.15.64:0/1081381642 conn(0x7f3ce00fc4d0 msgr2=0x7f3ce00fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 shutdown_connections 2024-08-19T01:31:49.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:49.986+0000 7f3ce6214640 1 -- 172.21.15.64:0/1081381642 wait complete. 2024-08-19T01:31:50.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:50 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.796 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-08-19T01:31:49.686508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651659,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":236,"num_read_kb":229,"num_write":341,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1185199,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361388,"kb_used_data":12212,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580436,"statfs":{"total":1151940427776,"available":1151570366464,"internally_reserved":0,"allocated":12505088,"data_stored":8065843,"data_compressed":390537,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":53840,"num_objects":3,"num_object_clones":0,"num_object_copies":11,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":95,"num_write":34,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4463,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595389+0000","last_change":"2024-08-19T01:30:49.539512+0000","last_active":"2024-08-19T01:31:35.595389+0000","last_peered":"2024-08-19T01:31:35.595389+0000","last_clean":"2024-08-19T01:31:35.595389+0000","last_became_active":"2024-08-19T01:30:49.539163+0000","last_became_peered":"2024-08-19T01:30:49.539163+0000","last_unstale":"2024-08-19T01:31:35.595389+0000","last_undegraded":"2024-08-19T01:31:35.595389+0000","last_fullsized":"2024-08-19T01:31:35.595389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:23.874916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868682+0000","last_change":"2024-08-19T01:30:51.561135+0000","last_active":"2024-08-19T01:31:35.868682+0000","last_peered":"2024-08-19T01:31:35.868682+0000","last_clean":"2024-08-19T01:31:35.868682+0000","last_became_active":"2024-08-19T01:30:51.560822+0000","last_became_peered":"2024-08-19T01:30:51.560822+0000","last_unstale":"2024-08-19T01:31:35.868682+0000","last_undegraded":"2024-08-19T01:31:35.868682+0000","last_fullsized":"2024-08-19T01:31:35.868682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:08:35.918895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597319+0000","last_change":"2024-08-19T01:30:58.595727+0000","last_active":"2024-08-19T01:31:35.597319+0000","last_peered":"2024-08-19T01:31:35.597319+0000","last_clean":"2024-08-19T01:31:35.597319+0000","last_became_active":"2024-08-19T01:30:58.595286+0000","last_became_peered":"2024-08-19T01:30:58.595286+0000","last_unstale":"2024-08-19T01:31:35.597319+0000","last_undegraded":"2024-08-19T01:31:35.597319+0000","last_fullsized":"2024-08-19T01:31:35.597319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:25:02.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868994+0000","last_change":"2024-08-19T01:30:49.575693+0000","last_active":"2024-08-19T01:31:35.868994+0000","last_peered":"2024-08-19T01:31:35.868994+0000","last_clean":"2024-08-19T01:31:35.868994+0000","last_became_active":"2024-08-19T01:30:49.575331+0000","last_became_peered":"2024-08-19T01:30:49.575331+0000","last_unstale":"2024-08-19T01:31:35.868994+0000","last_undegraded":"2024-08-19T01:31:35.868994+0000","last_fullsized":"2024-08-19T01:31:35.868994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:31:50.773160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598318+0000","last_change":"2024-08-19T01:30:51.562446+0000","last_active":"2024-08-19T01:31:35.598318+0000","last_peered":"2024-08-19T01:31:35.598318+0000","last_clean":"2024-08-19T01:31:35.598318+0000","last_became_active":"2024-08-19T01:30:51.561475+0000","last_became_peered":"2024-08-19T01:30:51.561475+0000","last_unstale":"2024-08-19T01:31:35.598318+0000","last_undegraded":"2024-08-19T01:31:35.598318+0000","last_fullsized":"2024-08-19T01:31:35.598318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:51:41.531247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599871+0000","last_change":"2024-08-19T01:30:58.591371+0000","last_active":"2024-08-19T01:31:35.599871+0000","last_peered":"2024-08-19T01:31:35.599871+0000","last_clean":"2024-08-19T01:31:35.599871+0000","last_became_active":"2024-08-19T01:30:58.590936+0000","last_became_peered":"2024-08-19T01:30:58.590936+0000","last_unstale":"2024-08-19T01:31:35.599871+0000","last_undegraded":"2024-08-19T01:31:35.599871+0000","last_fullsized":"2024-08-19T01:31:35.599871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:07:33.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.613588+0000","last_change":"2024-08-19T01:31:35.613145+0000","last_active":"2024-08-19T01:31:35.613588+0000","last_peered":"2024-08-19T01:31:35.613588+0000","last_clean":"2024-08-19T01:31:35.613588+0000","last_became_active":"2024-08-19T01:31:35.605087+0000","last_became_peered":"2024-08-19T01:31:35.605087+0000","last_unstale":"2024-08-19T01:31:35.613588+0000","last_undegraded":"2024-08-19T01:31:35.613588+0000","last_fullsized":"2024-08-19T01:31:35.613588+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:25:59.109707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596225+0000","last_change":"2024-08-19T01:30:51.555256+0000","last_active":"2024-08-19T01:31:35.596225+0000","last_peered":"2024-08-19T01:31:35.596225+0000","last_clean":"2024-08-19T01:31:35.596225+0000","last_became_active":"2024-08-19T01:30:51.554030+0000","last_became_peered":"2024-08-19T01:30:51.554030+0000","last_unstale":"2024-08-19T01:31:35.596225+0000","last_undegraded":"2024-08-19T01:31:35.596225+0000","last_fullsized":"2024-08-19T01:31:35.596225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:12:04.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600498+0000","last_change":"2024-08-19T01:30:58.589362+0000","last_active":"2024-08-19T01:31:35.600498+0000","last_peered":"2024-08-19T01:31:35.600498+0000","last_clean":"2024-08-19T01:31:35.600498+0000","last_became_active":"2024-08-19T01:30:58.588937+0000","last_became_peered":"2024-08-19T01:30:58.588937+0000","last_unstale":"2024-08-19T01:31:35.600498+0000","last_undegraded":"2024-08-19T01:31:35.600498+0000","last_fullsized":"2024-08-19T01:31:35.600498+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:31.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604267+0000","last_change":"2024-08-19T01:30:49.538597+0000","last_active":"2024-08-19T01:31:35.604267+0000","last_peered":"2024-08-19T01:31:35.604267+0000","last_clean":"2024-08-19T01:31:35.604267+0000","last_became_active":"2024-08-19T01:30:49.538228+0000","last_became_peered":"2024-08-19T01:30:49.538228+0000","last_unstale":"2024-08-19T01:31:35.604267+0000","last_undegraded":"2024-08-19T01:31:35.604267+0000","last_fullsized":"2024-08-19T01:31:35.604267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:31.350962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600891+0000","last_change":"2024-08-19T01:30:51.562829+0000","last_active":"2024-08-19T01:31:35.600891+0000","last_peered":"2024-08-19T01:31:35.600891+0000","last_clean":"2024-08-19T01:31:35.600891+0000","last_became_active":"2024-08-19T01:30:51.562144+0000","last_became_peered":"2024-08-19T01:30:51.562144+0000","last_unstale":"2024-08-19T01:31:35.600891+0000","last_undegraded":"2024-08-19T01:31:35.600891+0000","last_fullsized":"2024-08-19T01:31:35.600891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:17:09.184222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599645+0000","last_change":"2024-08-19T01:30:58.593178+0000","last_active":"2024-08-19T01:31:35.599645+0000","last_peered":"2024-08-19T01:31:35.599645+0000","last_clean":"2024-08-19T01:31:35.599645+0000","last_became_active":"2024-08-19T01:30:58.591247+0000","last_became_peered":"2024-08-19T01:30:58.591247+0000","last_unstale":"2024-08-19T01:31:35.599645+0000","last_undegraded":"2024-08-19T01:31:35.599645+0000","last_fullsized":"2024-08-19T01:31:35.599645+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:20:40.796135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604428+0000","last_change":"2024-08-19T01:30:51.562307+0000","last_active":"2024-08-19T01:31:35.604428+0000","last_peered":"2024-08-19T01:31:35.604428+0000","last_clean":"2024-08-19T01:31:35.604428+0000","last_became_active":"2024-08-19T01:30:51.561957+0000","last_became_peered":"2024-08-19T01:30:51.561957+0000","last_unstale":"2024-08-19T01:31:35.604428+0000","last_undegraded":"2024-08-19T01:31:35.604428+0000","last_fullsized":"2024-08-19T01:31:35.604428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:22.902317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.610093+0000","last_change":"2024-08-19T01:31:35.609612+0000","last_active":"2024-08-19T01:31:35.610093+0000","last_peered":"2024-08-19T01:31:35.610093+0000","last_clean":"2024-08-19T01:31:35.610093+0000","last_became_active":"2024-08-19T01:31:35.604969+0000","last_became_peered":"2024-08-19T01:31:35.604969+0000","last_unstale":"2024-08-19T01:31:35.610093+0000","last_undegraded":"2024-08-19T01:31:35.610093+0000","last_fullsized":"2024-08-19T01:31:35.610093+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:59:38.089119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600108+0000","last_change":"2024-08-19T01:30:58.588984+0000","last_active":"2024-08-19T01:31:35.600108+0000","last_peered":"2024-08-19T01:31:35.600108+0000","last_clean":"2024-08-19T01:31:35.600108+0000","last_became_active":"2024-08-19T01:30:58.588504+0000","last_became_peered":"2024-08-19T01:30:58.588504+0000","last_unstale":"2024-08-19T01:31:35.600108+0000","last_undegraded":"2024-08-19T01:31:35.600108+0000","last_fullsized":"2024-08-19T01:31:35.600108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:29:14.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869072+0000","last_change":"2024-08-19T01:30:51.553293+0000","last_active":"2024-08-19T01:31:35.869072+0000","last_peered":"2024-08-19T01:31:35.869072+0000","last_clean":"2024-08-19T01:31:35.869072+0000","last_became_active":"2024-08-19T01:30:51.552990+0000","last_became_peered":"2024-08-19T01:30:51.552990+0000","last_unstale":"2024-08-19T01:31:35.869072+0000","last_undegraded":"2024-08-19T01:31:35.869072+0000","last_fullsized":"2024-08-19T01:31:35.869072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:28:33.193014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597523+0000","last_change":"2024-08-19T01:30:49.537742+0000","last_active":"2024-08-19T01:31:35.597523+0000","last_peered":"2024-08-19T01:31:35.597523+0000","last_clean":"2024-08-19T01:31:35.597523+0000","last_became_active":"2024-08-19T01:30:49.537153+0000","last_became_peered":"2024-08-19T01:30:49.537153+0000","last_unstale":"2024-08-19T01:31:35.597523+0000","last_undegraded":"2024-08-19T01:31:35.597523+0000","last_fullsized":"2024-08-19T01:31:35.597523+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:25:03.650900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600493+0000","last_change":"2024-08-19T01:30:58.592348+0000","last_active":"2024-08-19T01:31:35.600493+0000","last_peered":"2024-08-19T01:31:35.600493+0000","last_clean":"2024-08-19T01:31:35.600493+0000","last_became_active":"2024-08-19T01:30:58.591641+0000","last_became_peered":"2024-08-19T01:30:58.591641+0000","last_unstale":"2024-08-19T01:31:35.600493+0000","last_undegraded":"2024-08-19T01:31:35.600493+0000","last_fullsized":"2024-08-19T01:31:35.600493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:39:17.542800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599683+0000","last_change":"2024-08-19T01:30:51.552126+0000","last_active":"2024-08-19T01:31:35.599683+0000","last_peered":"2024-08-19T01:31:35.599683+0000","last_clean":"2024-08-19T01:31:35.599683+0000","last_became_active":"2024-08-19T01:30:51.551795+0000","last_became_peered":"2024-08-19T01:30:51.551795+0000","last_unstale":"2024-08-19T01:31:35.599683+0000","last_undegraded":"2024-08-19T01:31:35.599683+0000","last_fullsized":"2024-08-19T01:31:35.599683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:22:54.705359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868723+0000","last_change":"2024-08-19T01:30:49.536659+0000","last_active":"2024-08-19T01:31:35.868723+0000","last_peered":"2024-08-19T01:31:35.868723+0000","last_clean":"2024-08-19T01:31:35.868723+0000","last_became_active":"2024-08-19T01:30:49.536366+0000","last_became_peered":"2024-08-19T01:30:49.536366+0000","last_unstale":"2024-08-19T01:31:35.868723+0000","last_undegraded":"2024-08-19T01:31:35.868723+0000","last_fullsized":"2024-08-19T01:31:35.868723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:58:52.547461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595100+0000","last_change":"2024-08-19T01:30:58.593724+0000","last_active":"2024-08-19T01:31:35.595100+0000","last_peered":"2024-08-19T01:31:35.595100+0000","last_clean":"2024-08-19T01:31:35.595100+0000","last_became_active":"2024-08-19T01:30:58.593383+0000","last_became_peered":"2024-08-19T01:30:58.593383+0000","last_unstale":"2024-08-19T01:31:35.595100+0000","last_undegraded":"2024-08-19T01:31:35.595100+0000","last_fullsized":"2024-08-19T01:31:35.595100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:54:26.424139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603836+0000","last_change":"2024-08-19T01:30:51.557682+0000","last_active":"2024-08-19T01:31:35.603836+0000","last_peered":"2024-08-19T01:31:35.603836+0000","last_clean":"2024-08-19T01:31:35.603836+0000","last_became_active":"2024-08-19T01:30:51.557351+0000","last_became_peered":"2024-08-19T01:30:51.557351+0000","last_unstale":"2024-08-19T01:31:35.603836+0000","last_undegraded":"2024-08-19T01:31:35.603836+0000","last_fullsized":"2024-08-19T01:31:35.603836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:41.484955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595958+0000","last_change":"2024-08-19T01:30:49.575996+0000","last_active":"2024-08-19T01:31:35.595958+0000","last_peered":"2024-08-19T01:31:35.595958+0000","last_clean":"2024-08-19T01:31:35.595958+0000","last_became_active":"2024-08-19T01:30:49.575607+0000","last_became_peered":"2024-08-19T01:30:49.575607+0000","last_unstale":"2024-08-19T01:31:35.595958+0000","last_undegraded":"2024-08-19T01:31:35.595958+0000","last_fullsized":"2024-08-19T01:31:35.595958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:20:33.467100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600113+0000","last_change":"2024-08-19T01:30:58.592629+0000","last_active":"2024-08-19T01:31:35.600113+0000","last_peered":"2024-08-19T01:31:35.600113+0000","last_clean":"2024-08-19T01:31:35.600113+0000","last_became_active":"2024-08-19T01:30:58.592122+0000","last_became_peered":"2024-08-19T01:30:58.592122+0000","last_unstale":"2024-08-19T01:31:35.600113+0000","last_undegraded":"2024-08-19T01:31:35.600113+0000","last_fullsized":"2024-08-19T01:31:35.600113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:19:09.846829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.746416+0000","last_change":"2024-08-19T01:31:35.746416+0000","last_active":"2024-08-19T01:31:35.746416+0000","last_peered":"2024-08-19T01:31:35.746416+0000","last_clean":"2024-08-19T01:31:35.746416+0000","last_became_active":"2024-08-19T01:31:35.746110+0000","last_became_peered":"2024-08-19T01:31:35.746110+0000","last_unstale":"2024-08-19T01:31:35.746416+0000","last_undegraded":"2024-08-19T01:31:35.746416+0000","last_fullsized":"2024-08-19T01:31:35.746416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:29:29.789785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600943+0000","last_change":"2024-08-19T01:30:49.711366+0000","last_active":"2024-08-19T01:31:35.600943+0000","last_peered":"2024-08-19T01:31:35.600943+0000","last_clean":"2024-08-19T01:31:35.600943+0000","last_became_active":"2024-08-19T01:30:49.711078+0000","last_became_peered":"2024-08-19T01:30:49.711078+0000","last_unstale":"2024-08-19T01:31:35.600943+0000","last_undegraded":"2024-08-19T01:31:35.600943+0000","last_fullsized":"2024-08-19T01:31:35.600943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:11:16.767867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597334+0000","last_change":"2024-08-19T01:30:58.591845+0000","last_active":"2024-08-19T01:31:35.597334+0000","last_peered":"2024-08-19T01:31:35.597334+0000","last_clean":"2024-08-19T01:31:35.597334+0000","last_became_active":"2024-08-19T01:30:58.591453+0000","last_became_peered":"2024-08-19T01:30:58.591453+0000","last_unstale":"2024-08-19T01:31:35.597334+0000","last_undegraded":"2024-08-19T01:31:35.597334+0000","last_fullsized":"2024-08-19T01:31:35.597334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:37:09.521514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869013+0000","last_change":"2024-08-19T01:30:51.563156+0000","last_active":"2024-08-19T01:31:35.869013+0000","last_peered":"2024-08-19T01:31:35.869013+0000","last_clean":"2024-08-19T01:31:35.869013+0000","last_became_active":"2024-08-19T01:30:51.562860+0000","last_became_peered":"2024-08-19T01:30:51.562860+0000","last_unstale":"2024-08-19T01:31:35.869013+0000","last_undegraded":"2024-08-19T01:31:35.869013+0000","last_fullsized":"2024-08-19T01:31:35.869013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:54:11.768833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:42.558913+0000","last_change":"2024-08-19T01:30:49.536648+0000","last_active":"2024-08-19T01:31:42.558913+0000","last_peered":"2024-08-19T01:31:42.558913+0000","last_clean":"2024-08-19T01:31:42.558913+0000","last_became_active":"2024-08-19T01:30:49.536080+0000","last_became_peered":"2024-08-19T01:30:49.536080+0000","last_unstale":"2024-08-19T01:31:42.558913+0000","last_undegraded":"2024-08-19T01:31:42.558913+0000","last_fullsized":"2024-08-19T01:31:42.558913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:03:46.804252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868291+0000","last_change":"2024-08-19T01:30:58.592253+0000","last_active":"2024-08-19T01:31:35.868291+0000","last_peered":"2024-08-19T01:31:35.868291+0000","last_clean":"2024-08-19T01:31:35.868291+0000","last_became_active":"2024-08-19T01:30:58.591911+0000","last_became_peered":"2024-08-19T01:30:58.591911+0000","last_unstale":"2024-08-19T01:31:35.868291+0000","last_undegraded":"2024-08-19T01:31:35.868291+0000","last_fullsized":"2024-08-19T01:31:35.868291+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:02:27.112597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745492+0000","last_change":"2024-08-19T01:30:51.547607+0000","last_active":"2024-08-19T01:31:35.745492+0000","last_peered":"2024-08-19T01:31:35.745492+0000","last_clean":"2024-08-19T01:31:35.745492+0000","last_became_active":"2024-08-19T01:30:51.547316+0000","last_became_peered":"2024-08-19T01:30:51.547316+0000","last_unstale":"2024-08-19T01:31:35.745492+0000","last_undegraded":"2024-08-19T01:31:35.745492+0000","last_fullsized":"2024-08-19T01:31:35.745492+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:48:58.300508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604727+0000","last_change":"2024-08-19T01:30:49.576942+0000","last_active":"2024-08-19T01:31:35.604727+0000","last_peered":"2024-08-19T01:31:35.604727+0000","last_clean":"2024-08-19T01:31:35.604727+0000","last_became_active":"2024-08-19T01:30:49.576528+0000","last_became_peered":"2024-08-19T01:30:49.576528+0000","last_unstale":"2024-08-19T01:31:35.604727+0000","last_undegraded":"2024-08-19T01:31:35.604727+0000","last_fullsized":"2024-08-19T01:31:35.604727+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:23:53.259395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600274+0000","last_change":"2024-08-19T01:30:58.591577+0000","last_active":"2024-08-19T01:31:35.600274+0000","last_peered":"2024-08-19T01:31:35.600274+0000","last_clean":"2024-08-19T01:31:35.600274+0000","last_became_active":"2024-08-19T01:30:58.591181+0000","last_became_peered":"2024-08-19T01:30:58.591181+0000","last_unstale":"2024-08-19T01:31:35.600274+0000","last_undegraded":"2024-08-19T01:31:35.600274+0000","last_fullsized":"2024-08-19T01:31:35.600274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:02:59.682147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598931+0000","last_change":"2024-08-19T01:30:51.548556+0000","last_active":"2024-08-19T01:31:35.598931+0000","last_peered":"2024-08-19T01:31:35.598931+0000","last_clean":"2024-08-19T01:31:35.598931+0000","last_became_active":"2024-08-19T01:30:51.547838+0000","last_became_peered":"2024-08-19T01:30:51.547838+0000","last_unstale":"2024-08-19T01:31:35.598931+0000","last_undegraded":"2024-08-19T01:31:35.598931+0000","last_fullsized":"2024-08-19T01:31:35.598931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:08:49.671964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595172+0000","last_change":"2024-08-19T01:30:49.576799+0000","last_active":"2024-08-19T01:31:35.595172+0000","last_peered":"2024-08-19T01:31:35.595172+0000","last_clean":"2024-08-19T01:31:35.595172+0000","last_became_active":"2024-08-19T01:30:49.576385+0000","last_became_peered":"2024-08-19T01:30:49.576385+0000","last_unstale":"2024-08-19T01:31:35.595172+0000","last_undegraded":"2024-08-19T01:31:35.595172+0000","last_fullsized":"2024-08-19T01:31:35.595172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:34:18.287518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603305+0000","last_change":"2024-08-19T01:30:58.593703+0000","last_active":"2024-08-19T01:31:35.603305+0000","last_peered":"2024-08-19T01:31:35.603305+0000","last_clean":"2024-08-19T01:31:35.603305+0000","last_became_active":"2024-08-19T01:30:58.593303+0000","last_became_peered":"2024-08-19T01:30:58.593303+0000","last_unstale":"2024-08-19T01:31:35.603305+0000","last_undegraded":"2024-08-19T01:31:35.603305+0000","last_fullsized":"2024-08-19T01:31:35.603305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:12:43.359262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601173+0000","last_change":"2024-08-19T01:31:35.601173+0000","last_active":"2024-08-19T01:31:35.601173+0000","last_peered":"2024-08-19T01:31:35.601173+0000","last_clean":"2024-08-19T01:31:35.601173+0000","last_became_active":"2024-08-19T01:31:35.600742+0000","last_became_peered":"2024-08-19T01:31:35.600742+0000","last_unstale":"2024-08-19T01:31:35.601173+0000","last_undegraded":"2024-08-19T01:31:35.601173+0000","last_fullsized":"2024-08-19T01:31:35.601173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:36:48.977124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604573+0000","last_change":"2024-08-19T01:30:49.540070+0000","last_active":"2024-08-19T01:31:35.604573+0000","last_peered":"2024-08-19T01:31:35.604573+0000","last_clean":"2024-08-19T01:31:35.604573+0000","last_became_active":"2024-08-19T01:30:49.539706+0000","last_became_peered":"2024-08-19T01:30:49.539706+0000","last_unstale":"2024-08-19T01:31:35.604573+0000","last_undegraded":"2024-08-19T01:31:35.604573+0000","last_fullsized":"2024-08-19T01:31:35.604573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:14:15.813670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745319+0000","last_change":"2024-08-19T01:30:58.590766+0000","last_active":"2024-08-19T01:31:35.745319+0000","last_peered":"2024-08-19T01:31:35.745319+0000","last_clean":"2024-08-19T01:31:35.745319+0000","last_became_active":"2024-08-19T01:30:58.590415+0000","last_became_peered":"2024-08-19T01:30:58.590415+0000","last_unstale":"2024-08-19T01:31:35.745319+0000","last_undegraded":"2024-08-19T01:31:35.745319+0000","last_fullsized":"2024-08-19T01:31:35.745319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:48:59.261086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.602761+0000","last_change":"2024-08-19T01:31:35.602332+0000","last_active":"2024-08-19T01:31:35.602761+0000","last_peered":"2024-08-19T01:31:35.602761+0000","last_clean":"2024-08-19T01:31:35.602761+0000","last_became_active":"2024-08-19T01:31:35.602007+0000","last_became_peered":"2024-08-19T01:31:35.602007+0000","last_unstale":"2024-08-19T01:31:35.602761+0000","last_undegraded":"2024-08-19T01:31:35.602761+0000","last_fullsized":"2024-08-19T01:31:35.602761+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:17:47.694567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:44.614804+0000","last_change":"2024-08-19T01:30:49.575995+0000","last_active":"2024-08-19T01:31:44.614804+0000","last_peered":"2024-08-19T01:31:44.614804+0000","last_clean":"2024-08-19T01:31:44.614804+0000","last_became_active":"2024-08-19T01:30:49.575249+0000","last_became_peered":"2024-08-19T01:30:49.575249+0000","last_unstale":"2024-08-19T01:31:44.614804+0000","last_undegraded":"2024-08-19T01:31:44.614804+0000","last_fullsized":"2024-08-19T01:31:44.614804+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:39:05.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745536+0000","last_change":"2024-08-19T01:30:58.598812+0000","last_active":"2024-08-19T01:31:35.745536+0000","last_peered":"2024-08-19T01:31:35.745536+0000","last_clean":"2024-08-19T01:31:35.745536+0000","last_became_active":"2024-08-19T01:30:58.598434+0000","last_became_peered":"2024-08-19T01:30:58.598434+0000","last_unstale":"2024-08-19T01:31:35.745536+0000","last_undegraded":"2024-08-19T01:31:35.745536+0000","last_fullsized":"2024-08-19T01:31:35.745536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:15:19.381503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744921+0000","last_change":"2024-08-19T01:30:51.549158+0000","last_active":"2024-08-19T01:31:35.744921+0000","last_peered":"2024-08-19T01:31:35.744921+0000","last_clean":"2024-08-19T01:31:35.744921+0000","last_became_active":"2024-08-19T01:30:51.548854+0000","last_became_peered":"2024-08-19T01:30:51.548854+0000","last_unstale":"2024-08-19T01:31:35.744921+0000","last_undegraded":"2024-08-19T01:31:35.744921+0000","last_fullsized":"2024-08-19T01:31:35.744921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:35.661758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604219+0000","last_change":"2024-08-19T01:30:49.538282+0000","last_active":"2024-08-19T01:31:35.604219+0000","last_peered":"2024-08-19T01:31:35.604219+0000","last_clean":"2024-08-19T01:31:35.604219+0000","last_became_active":"2024-08-19T01:30:49.537918+0000","last_became_peered":"2024-08-19T01:30:49.537918+0000","last_unstale":"2024-08-19T01:31:35.604219+0000","last_undegraded":"2024-08-19T01:31:35.604219+0000","last_fullsized":"2024-08-19T01:31:35.604219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:27.886958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601012+0000","last_change":"2024-08-19T01:30:58.591153+0000","last_active":"2024-08-19T01:31:35.601012+0000","last_peered":"2024-08-19T01:31:35.601012+0000","last_clean":"2024-08-19T01:31:35.601012+0000","last_became_active":"2024-08-19T01:30:58.590738+0000","last_became_peered":"2024-08-19T01:30:58.590738+0000","last_unstale":"2024-08-19T01:31:35.601012+0000","last_undegraded":"2024-08-19T01:31:35.601012+0000","last_fullsized":"2024-08-19T01:31:35.601012+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:29.073825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868743+0000","last_change":"2024-08-19T01:30:51.554878+0000","last_active":"2024-08-19T01:31:35.868743+0000","last_peered":"2024-08-19T01:31:35.868743+0000","last_clean":"2024-08-19T01:31:35.868743+0000","last_became_active":"2024-08-19T01:30:51.553743+0000","last_became_peered":"2024-08-19T01:30:51.553743+0000","last_unstale":"2024-08-19T01:31:35.868743+0000","last_undegraded":"2024-08-19T01:31:35.868743+0000","last_fullsized":"2024-08-19T01:31:35.868743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:08:41.932641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600867+0000","last_change":"2024-08-19T01:30:49.542043+0000","last_active":"2024-08-19T01:31:35.600867+0000","last_peered":"2024-08-19T01:31:35.600867+0000","last_clean":"2024-08-19T01:31:35.600867+0000","last_became_active":"2024-08-19T01:30:49.541551+0000","last_became_peered":"2024-08-19T01:30:49.541551+0000","last_unstale":"2024-08-19T01:31:35.600867+0000","last_undegraded":"2024-08-19T01:31:35.600867+0000","last_fullsized":"2024-08-19T01:31:35.600867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:46:14.781516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":364,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604258+0000","last_change":"2024-08-19T01:29:41.574391+0000","last_active":"2024-08-19T01:31:35.604258+0000","last_peered":"2024-08-19T01:31:35.604258+0000","last_clean":"2024-08-19T01:31:35.604258+0000","last_became_active":"2024-08-19T01:29:41.573885+0000","last_became_peered":"2024-08-19T01:29:41.573885+0000","last_unstale":"2024-08-19T01:31:35.604258+0000","last_undegraded":"2024-08-19T01:31:35.604258+0000","last_fullsized":"2024-08-19T01:31:35.604258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_clean_scrub_stamp":"2024-08-19T01:26:37.926225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:55:38.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:58.598415+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:58.598086+0000","last_became_peered":"2024-08-19T01:30:58.598086+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:21:37.338869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605022+0000","last_change":"2024-08-19T01:30:51.560407+0000","last_active":"2024-08-19T01:31:35.605022+0000","last_peered":"2024-08-19T01:31:35.605022+0000","last_clean":"2024-08-19T01:31:35.605022+0000","last_became_active":"2024-08-19T01:30:51.559881+0000","last_became_peered":"2024-08-19T01:30:51.559881+0000","last_unstale":"2024-08-19T01:31:35.605022+0000","last_undegraded":"2024-08-19T01:31:35.605022+0000","last_fullsized":"2024-08-19T01:31:35.605022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:24:37.928254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597862+0000","last_change":"2024-08-19T01:30:49.711171+0000","last_active":"2024-08-19T01:31:35.597862+0000","last_peered":"2024-08-19T01:31:35.597862+0000","last_clean":"2024-08-19T01:31:35.597862+0000","last_became_active":"2024-08-19T01:30:49.710707+0000","last_became_peered":"2024-08-19T01:30:49.710707+0000","last_unstale":"2024-08-19T01:31:35.597862+0000","last_undegraded":"2024-08-19T01:31:35.597862+0000","last_fullsized":"2024-08-19T01:31:35.597862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:58:20.924502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595955+0000","last_change":"2024-08-19T01:30:58.590829+0000","last_active":"2024-08-19T01:31:35.595955+0000","last_peered":"2024-08-19T01:31:35.595955+0000","last_clean":"2024-08-19T01:31:35.595955+0000","last_became_active":"2024-08-19T01:30:58.590460+0000","last_became_peered":"2024-08-19T01:30:58.590460+0000","last_unstale":"2024-08-19T01:31:35.595955+0000","last_undegraded":"2024-08-19T01:31:35.595955+0000","last_fullsized":"2024-08-19T01:31:35.595955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:04.118550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604645+0000","last_change":"2024-08-19T01:30:51.564990+0000","last_active":"2024-08-19T01:31:35.604645+0000","last_peered":"2024-08-19T01:31:35.604645+0000","last_clean":"2024-08-19T01:31:35.604645+0000","last_became_active":"2024-08-19T01:30:51.564653+0000","last_became_peered":"2024-08-19T01:30:51.564653+0000","last_unstale":"2024-08-19T01:31:35.604645+0000","last_undegraded":"2024-08-19T01:31:35.604645+0000","last_fullsized":"2024-08-19T01:31:35.604645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:15:41.154376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601128+0000","last_change":"2024-08-19T01:30:49.537637+0000","last_active":"2024-08-19T01:31:35.601128+0000","last_peered":"2024-08-19T01:31:35.601128+0000","last_clean":"2024-08-19T01:31:35.601128+0000","last_became_active":"2024-08-19T01:30:49.537277+0000","last_became_peered":"2024-08-19T01:30:49.537277+0000","last_unstale":"2024-08-19T01:31:35.601128+0000","last_undegraded":"2024-08-19T01:31:35.601128+0000","last_fullsized":"2024-08-19T01:31:35.601128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:13.547597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598003+0000","last_change":"2024-08-19T01:30:58.594675+0000","last_active":"2024-08-19T01:31:35.598003+0000","last_peered":"2024-08-19T01:31:35.598003+0000","last_clean":"2024-08-19T01:31:35.598003+0000","last_became_active":"2024-08-19T01:30:58.594323+0000","last_became_peered":"2024-08-19T01:30:58.594323+0000","last_unstale":"2024-08-19T01:31:35.598003+0000","last_undegraded":"2024-08-19T01:31:35.598003+0000","last_fullsized":"2024-08-19T01:31:35.598003+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:21:05.447106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605468+0000","last_change":"2024-08-19T01:30:51.565014+0000","last_active":"2024-08-19T01:31:35.605468+0000","last_peered":"2024-08-19T01:31:35.605468+0000","last_clean":"2024-08-19T01:31:35.605468+0000","last_became_active":"2024-08-19T01:30:51.564672+0000","last_became_peered":"2024-08-19T01:30:51.564672+0000","last_unstale":"2024-08-19T01:31:35.605468+0000","last_undegraded":"2024-08-19T01:31:35.605468+0000","last_fullsized":"2024-08-19T01:31:35.605468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:56:07.404042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745571+0000","last_change":"2024-08-19T01:30:49.538423+0000","last_active":"2024-08-19T01:31:35.745571+0000","last_peered":"2024-08-19T01:31:35.745571+0000","last_clean":"2024-08-19T01:31:35.745571+0000","last_became_active":"2024-08-19T01:30:49.538049+0000","last_became_peered":"2024-08-19T01:30:49.538049+0000","last_unstale":"2024-08-19T01:31:35.745571+0000","last_undegraded":"2024-08-19T01:31:35.745571+0000","last_fullsized":"2024-08-19T01:31:35.745571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:27:41.449742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599789+0000","last_change":"2024-08-19T01:30:58.590963+0000","last_active":"2024-08-19T01:31:35.599789+0000","last_peered":"2024-08-19T01:31:35.599789+0000","last_clean":"2024-08-19T01:31:35.599789+0000","last_became_active":"2024-08-19T01:30:58.590584+0000","last_became_peered":"2024-08-19T01:30:58.590584+0000","last_unstale":"2024-08-19T01:31:35.599789+0000","last_undegraded":"2024-08-19T01:31:35.599789+0000","last_fullsized":"2024-08-19T01:31:35.599789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:37.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599809+0000","last_change":"2024-08-19T01:30:51.561298+0000","last_active":"2024-08-19T01:31:35.599809+0000","last_peered":"2024-08-19T01:31:35.599809+0000","last_clean":"2024-08-19T01:31:35.599809+0000","last_became_active":"2024-08-19T01:30:51.560919+0000","last_became_peered":"2024-08-19T01:30:51.560919+0000","last_unstale":"2024-08-19T01:31:35.599809+0000","last_undegraded":"2024-08-19T01:31:35.599809+0000","last_fullsized":"2024-08-19T01:31:35.599809+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:04:39.808838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604897+0000","last_change":"2024-08-19T01:30:49.541968+0000","last_active":"2024-08-19T01:31:35.604897+0000","last_peered":"2024-08-19T01:31:35.604897+0000","last_clean":"2024-08-19T01:31:35.604897+0000","last_became_active":"2024-08-19T01:30:49.541512+0000","last_became_peered":"2024-08-19T01:30:49.541512+0000","last_unstale":"2024-08-19T01:31:35.604897+0000","last_undegraded":"2024-08-19T01:31:35.604897+0000","last_fullsized":"2024-08-19T01:31:35.604897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:34:28.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600022+0000","last_change":"2024-08-19T01:30:58.594776+0000","last_active":"2024-08-19T01:31:35.600022+0000","last_peered":"2024-08-19T01:31:35.600022+0000","last_clean":"2024-08-19T01:31:35.600022+0000","last_became_active":"2024-08-19T01:30:58.594414+0000","last_became_peered":"2024-08-19T01:30:58.594414+0000","last_unstale":"2024-08-19T01:31:35.600022+0000","last_undegraded":"2024-08-19T01:31:35.600022+0000","last_fullsized":"2024-08-19T01:31:35.600022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:32:20.121504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600072+0000","last_change":"2024-08-19T01:30:51.556696+0000","last_active":"2024-08-19T01:31:35.600072+0000","last_peered":"2024-08-19T01:31:35.600072+0000","last_clean":"2024-08-19T01:31:35.600072+0000","last_became_active":"2024-08-19T01:30:51.555873+0000","last_became_peered":"2024-08-19T01:30:51.555873+0000","last_unstale":"2024-08-19T01:31:35.600072+0000","last_undegraded":"2024-08-19T01:31:35.600072+0000","last_fullsized":"2024-08-19T01:31:35.600072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:50:27.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603398+0000","last_change":"2024-08-19T01:30:49.576925+0000","last_active":"2024-08-19T01:31:35.603398+0000","last_peered":"2024-08-19T01:31:35.603398+0000","last_clean":"2024-08-19T01:31:35.603398+0000","last_became_active":"2024-08-19T01:30:49.576509+0000","last_became_peered":"2024-08-19T01:30:49.576509+0000","last_unstale":"2024-08-19T01:31:35.603398+0000","last_undegraded":"2024-08-19T01:31:35.603398+0000","last_fullsized":"2024-08-19T01:31:35.603398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:43:01.887020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745511+0000","last_change":"2024-08-19T01:30:58.590940+0000","last_active":"2024-08-19T01:31:35.745511+0000","last_peered":"2024-08-19T01:31:35.745511+0000","last_clean":"2024-08-19T01:31:35.745511+0000","last_became_active":"2024-08-19T01:30:58.590565+0000","last_became_peered":"2024-08-19T01:30:58.590565+0000","last_unstale":"2024-08-19T01:31:35.745511+0000","last_undegraded":"2024-08-19T01:31:35.745511+0000","last_fullsized":"2024-08-19T01:31:35.745511+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:38:31.345290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603937+0000","last_change":"2024-08-19T01:30:51.559622+0000","last_active":"2024-08-19T01:31:35.603937+0000","last_peered":"2024-08-19T01:31:35.603937+0000","last_clean":"2024-08-19T01:31:35.603937+0000","last_became_active":"2024-08-19T01:30:51.559263+0000","last_became_peered":"2024-08-19T01:30:51.559263+0000","last_unstale":"2024-08-19T01:31:35.603937+0000","last_undegraded":"2024-08-19T01:31:35.603937+0000","last_fullsized":"2024-08-19T01:31:35.603937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:11:25.758915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:44.615644+0000","last_change":"2024-08-19T01:30:49.540102+0000","last_active":"2024-08-19T01:31:44.615644+0000","last_peered":"2024-08-19T01:31:44.615644+0000","last_clean":"2024-08-19T01:31:44.615644+0000","last_became_active":"2024-08-19T01:30:49.539740+0000","last_became_peered":"2024-08-19T01:30:49.539740+0000","last_unstale":"2024-08-19T01:31:44.615644+0000","last_undegraded":"2024-08-19T01:31:44.615644+0000","last_fullsized":"2024-08-19T01:31:44.615644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:47:21.684925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595191+0000","last_change":"2024-08-19T01:30:58.595748+0000","last_active":"2024-08-19T01:31:35.595191+0000","last_peered":"2024-08-19T01:31:35.595191+0000","last_clean":"2024-08-19T01:31:35.595191+0000","last_became_active":"2024-08-19T01:30:58.595396+0000","last_became_peered":"2024-08-19T01:30:58.595396+0000","last_unstale":"2024-08-19T01:31:35.595191+0000","last_undegraded":"2024-08-19T01:31:35.595191+0000","last_fullsized":"2024-08-19T01:31:35.595191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:58:29.113687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604652+0000","last_change":"2024-08-19T01:30:51.559159+0000","last_active":"2024-08-19T01:31:35.604652+0000","last_peered":"2024-08-19T01:31:35.604652+0000","last_clean":"2024-08-19T01:31:35.604652+0000","last_became_active":"2024-08-19T01:30:51.558226+0000","last_became_peered":"2024-08-19T01:30:51.558226+0000","last_unstale":"2024-08-19T01:31:35.604652+0000","last_undegraded":"2024-08-19T01:31:35.604652+0000","last_fullsized":"2024-08-19T01:31:35.604652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:12:54.447301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600514+0000","last_change":"2024-08-19T01:30:49.540806+0000","last_active":"2024-08-19T01:31:35.600514+0000","last_peered":"2024-08-19T01:31:35.600514+0000","last_clean":"2024-08-19T01:31:35.600514+0000","last_became_active":"2024-08-19T01:30:49.540439+0000","last_became_peered":"2024-08-19T01:30:49.540439+0000","last_unstale":"2024-08-19T01:31:35.600514+0000","last_undegraded":"2024-08-19T01:31:35.600514+0000","last_fullsized":"2024-08-19T01:31:35.600514+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:40.088842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599613+0000","last_change":"2024-08-19T01:30:58.592661+0000","last_active":"2024-08-19T01:31:35.599613+0000","last_peered":"2024-08-19T01:31:35.599613+0000","last_clean":"2024-08-19T01:31:35.599613+0000","last_became_active":"2024-08-19T01:30:58.592066+0000","last_became_peered":"2024-08-19T01:30:58.592066+0000","last_unstale":"2024-08-19T01:31:35.599613+0000","last_undegraded":"2024-08-19T01:31:35.599613+0000","last_fullsized":"2024-08-19T01:31:35.599613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:35.039682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745099+0000","last_change":"2024-08-19T01:30:51.558497+0000","last_active":"2024-08-19T01:31:35.745099+0000","last_peered":"2024-08-19T01:31:35.745099+0000","last_clean":"2024-08-19T01:31:35.745099+0000","last_became_active":"2024-08-19T01:30:51.558185+0000","last_became_peered":"2024-08-19T01:30:51.558185+0000","last_unstale":"2024-08-19T01:31:35.745099+0000","last_undegraded":"2024-08-19T01:31:35.745099+0000","last_fullsized":"2024-08-19T01:31:35.745099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:30:42.766269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597725+0000","last_change":"2024-08-19T01:30:49.539112+0000","last_active":"2024-08-19T01:31:35.597725+0000","last_peered":"2024-08-19T01:31:35.597725+0000","last_clean":"2024-08-19T01:31:35.597725+0000","last_became_active":"2024-08-19T01:30:49.538798+0000","last_became_peered":"2024-08-19T01:30:49.538798+0000","last_unstale":"2024-08-19T01:31:35.597725+0000","last_undegraded":"2024-08-19T01:31:35.597725+0000","last_fullsized":"2024-08-19T01:31:35.597725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:10:58.977041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600778+0000","last_change":"2024-08-19T01:30:58.590727+0000","last_active":"2024-08-19T01:31:35.600778+0000","last_peered":"2024-08-19T01:31:35.600778+0000","last_clean":"2024-08-19T01:31:35.600778+0000","last_became_active":"2024-08-19T01:30:58.590208+0000","last_became_peered":"2024-08-19T01:30:58.590208+0000","last_unstale":"2024-08-19T01:31:35.600778+0000","last_undegraded":"2024-08-19T01:31:35.600778+0000","last_fullsized":"2024-08-19T01:31:35.600778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:36.422915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604177+0000","last_change":"2024-08-19T01:30:51.565113+0000","last_active":"2024-08-19T01:31:35.604177+0000","last_peered":"2024-08-19T01:31:35.604177+0000","last_clean":"2024-08-19T01:31:35.604177+0000","last_became_active":"2024-08-19T01:30:51.564769+0000","last_became_peered":"2024-08-19T01:30:51.564769+0000","last_unstale":"2024-08-19T01:31:35.604177+0000","last_undegraded":"2024-08-19T01:31:35.604177+0000","last_fullsized":"2024-08-19T01:31:35.604177+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:13:28.373114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'29","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:47.624786+0000","last_change":"2024-08-19T01:31:35.610319+0000","last_active":"2024-08-19T01:31:47.624786+0000","last_peered":"2024-08-19T01:31:47.624786+0000","last_clean":"2024-08-19T01:31:47.624786+0000","last_became_active":"2024-08-19T01:31:35.602527+0000","last_became_peered":"2024-08-19T01:31:35.602527+0000","last_unstale":"2024-08-19T01:31:47.624786+0000","last_undegraded":"2024-08-19T01:31:47.624786+0000","last_fullsized":"2024-08-19T01:31:47.624786+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:50:05.511332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56667,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":67,"num_write":30,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3841,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594820+0000","last_change":"2024-08-19T01:30:58.594836+0000","last_active":"2024-08-19T01:31:35.594820+0000","last_peered":"2024-08-19T01:31:35.594820+0000","last_clean":"2024-08-19T01:31:35.594820+0000","last_became_active":"2024-08-19T01:30:58.594441+0000","last_became_peered":"2024-08-19T01:30:58.594441+0000","last_unstale":"2024-08-19T01:31:35.594820+0000","last_undegraded":"2024-08-19T01:31:35.594820+0000","last_fullsized":"2024-08-19T01:31:35.594820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:29:16.482181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596226+0000","last_change":"2024-08-19T01:30:51.562567+0000","last_active":"2024-08-19T01:31:35.596226+0000","last_peered":"2024-08-19T01:31:35.596226+0000","last_clean":"2024-08-19T01:31:35.596226+0000","last_became_active":"2024-08-19T01:30:51.562211+0000","last_became_peered":"2024-08-19T01:30:51.562211+0000","last_unstale":"2024-08-19T01:31:35.596226+0000","last_undegraded":"2024-08-19T01:31:35.596226+0000","last_fullsized":"2024-08-19T01:31:35.596226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:58.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601097+0000","last_change":"2024-08-19T01:30:49.575957+0000","last_active":"2024-08-19T01:31:35.601097+0000","last_peered":"2024-08-19T01:31:35.601097+0000","last_clean":"2024-08-19T01:31:35.601097+0000","last_became_active":"2024-08-19T01:30:49.575570+0000","last_became_peered":"2024-08-19T01:30:49.575570+0000","last_unstale":"2024-08-19T01:31:35.601097+0000","last_undegraded":"2024-08-19T01:31:35.601097+0000","last_fullsized":"2024-08-19T01:31:35.601097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:47.038113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599852+0000","last_change":"2024-08-19T01:30:58.589830+0000","last_active":"2024-08-19T01:31:35.599852+0000","last_peered":"2024-08-19T01:31:35.599852+0000","last_clean":"2024-08-19T01:31:35.599852+0000","last_became_active":"2024-08-19T01:30:58.589292+0000","last_became_peered":"2024-08-19T01:30:58.589292+0000","last_unstale":"2024-08-19T01:31:35.599852+0000","last_undegraded":"2024-08-19T01:31:35.599852+0000","last_fullsized":"2024-08-19T01:31:35.599852+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:14.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.606449+0000","last_change":"2024-08-19T01:31:35.606449+0000","last_active":"2024-08-19T01:31:35.606449+0000","last_peered":"2024-08-19T01:31:35.606449+0000","last_clean":"2024-08-19T01:31:35.606449+0000","last_became_active":"2024-08-19T01:31:35.606105+0000","last_became_peered":"2024-08-19T01:31:35.606105+0000","last_unstale":"2024-08-19T01:31:35.606449+0000","last_undegraded":"2024-08-19T01:31:35.606449+0000","last_fullsized":"2024-08-19T01:31:35.606449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:36:21.834763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605342+0000","last_change":"2024-08-19T01:30:49.542441+0000","last_active":"2024-08-19T01:31:35.605342+0000","last_peered":"2024-08-19T01:31:35.605342+0000","last_clean":"2024-08-19T01:31:35.605342+0000","last_became_active":"2024-08-19T01:30:49.542106+0000","last_became_peered":"2024-08-19T01:30:49.542106+0000","last_unstale":"2024-08-19T01:31:35.605342+0000","last_undegraded":"2024-08-19T01:31:35.605342+0000","last_fullsized":"2024-08-19T01:31:35.605342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:21:01.956282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594574+0000","last_change":"2024-08-19T01:30:58.588899+0000","last_active":"2024-08-19T01:31:35.594574+0000","last_peered":"2024-08-19T01:31:35.594574+0000","last_clean":"2024-08-19T01:31:35.594574+0000","last_became_active":"2024-08-19T01:30:58.588521+0000","last_became_peered":"2024-08-19T01:30:58.588521+0000","last_unstale":"2024-08-19T01:31:35.594574+0000","last_undegraded":"2024-08-19T01:31:35.594574+0000","last_fullsized":"2024-08-19T01:31:35.594574+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:43:59.485456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596747+0000","last_change":"2024-08-19T01:30:51.557248+0000","last_active":"2024-08-19T01:31:35.596747+0000","last_peered":"2024-08-19T01:31:35.596747+0000","last_clean":"2024-08-19T01:31:35.596747+0000","last_became_active":"2024-08-19T01:30:51.556448+0000","last_became_peered":"2024-08-19T01:30:51.556448+0000","last_unstale":"2024-08-19T01:31:35.596747+0000","last_undegraded":"2024-08-19T01:31:35.596747+0000","last_fullsized":"2024-08-19T01:31:35.596747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:19:19.587955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:46.623075+0000","last_change":"2024-08-19T01:30:49.540360+0000","last_active":"2024-08-19T01:31:46.623075+0000","last_peered":"2024-08-19T01:31:46.623075+0000","last_clean":"2024-08-19T01:31:46.623075+0000","last_became_active":"2024-08-19T01:30:49.539993+0000","last_became_peered":"2024-08-19T01:30:49.539993+0000","last_unstale":"2024-08-19T01:31:46.623075+0000","last_undegraded":"2024-08-19T01:31:46.623075+0000","last_fullsized":"2024-08-19T01:31:46.623075+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:05:28.308441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868467+0000","last_change":"2024-08-19T01:30:58.592387+0000","last_active":"2024-08-19T01:31:35.868467+0000","last_peered":"2024-08-19T01:31:35.868467+0000","last_clean":"2024-08-19T01:31:35.868467+0000","last_became_active":"2024-08-19T01:30:58.592031+0000","last_became_peered":"2024-08-19T01:30:58.592031+0000","last_unstale":"2024-08-19T01:31:35.868467+0000","last_undegraded":"2024-08-19T01:31:35.868467+0000","last_fullsized":"2024-08-19T01:31:35.868467+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:31:23.159012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598112+0000","last_change":"2024-08-19T01:30:58.595650+0000","last_active":"2024-08-19T01:31:35.598112+0000","last_peered":"2024-08-19T01:31:35.598112+0000","last_clean":"2024-08-19T01:31:35.598112+0000","last_became_active":"2024-08-19T01:30:58.595304+0000","last_became_peered":"2024-08-19T01:30:58.595304+0000","last_unstale":"2024-08-19T01:31:35.598112+0000","last_undegraded":"2024-08-19T01:31:35.598112+0000","last_fullsized":"2024-08-19T01:31:35.598112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:23:53.517355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594771+0000","last_change":"2024-08-19T01:30:51.548683+0000","last_active":"2024-08-19T01:31:35.594771+0000","last_peered":"2024-08-19T01:31:35.594771+0000","last_clean":"2024-08-19T01:31:35.594771+0000","last_became_active":"2024-08-19T01:30:51.548382+0000","last_became_peered":"2024-08-19T01:30:51.548382+0000","last_unstale":"2024-08-19T01:31:35.594771+0000","last_undegraded":"2024-08-19T01:31:35.594771+0000","last_fullsized":"2024-08-19T01:31:35.594771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:46:37.551017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601362+0000","last_change":"2024-08-19T01:30:49.540136+0000","last_active":"2024-08-19T01:31:35.601362+0000","last_peered":"2024-08-19T01:31:35.601362+0000","last_clean":"2024-08-19T01:31:35.601362+0000","last_became_active":"2024-08-19T01:30:49.539680+0000","last_became_peered":"2024-08-19T01:30:49.539680+0000","last_unstale":"2024-08-19T01:31:35.601362+0000","last_undegraded":"2024-08-19T01:31:35.601362+0000","last_fullsized":"2024-08-19T01:31:35.601362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:25.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599759+0000","last_change":"2024-08-19T01:30:58.597130+0000","last_active":"2024-08-19T01:31:35.599759+0000","last_peered":"2024-08-19T01:31:35.599759+0000","last_clean":"2024-08-19T01:31:35.599759+0000","last_became_active":"2024-08-19T01:30:58.596834+0000","last_became_peered":"2024-08-19T01:30:58.596834+0000","last_unstale":"2024-08-19T01:31:35.599759+0000","last_undegraded":"2024-08-19T01:31:35.599759+0000","last_fullsized":"2024-08-19T01:31:35.599759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:13:10.441421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868410+0000","last_change":"2024-08-19T01:30:51.555575+0000","last_active":"2024-08-19T01:31:35.868410+0000","last_peered":"2024-08-19T01:31:35.868410+0000","last_clean":"2024-08-19T01:31:35.868410+0000","last_became_active":"2024-08-19T01:30:51.555309+0000","last_became_peered":"2024-08-19T01:30:51.555309+0000","last_unstale":"2024-08-19T01:31:35.868410+0000","last_undegraded":"2024-08-19T01:31:35.868410+0000","last_fullsized":"2024-08-19T01:31:35.868410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:11:25.515286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744964+0000","last_change":"2024-08-19T01:30:49.536844+0000","last_active":"2024-08-19T01:31:35.744964+0000","last_peered":"2024-08-19T01:31:35.744964+0000","last_clean":"2024-08-19T01:31:35.744964+0000","last_became_active":"2024-08-19T01:30:49.536496+0000","last_became_peered":"2024-08-19T01:30:49.536496+0000","last_unstale":"2024-08-19T01:31:35.744964+0000","last_undegraded":"2024-08-19T01:31:35.744964+0000","last_fullsized":"2024-08-19T01:31:35.744964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:04.917188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598260+0000","last_change":"2024-08-19T01:30:49.539356+0000","last_active":"2024-08-19T01:31:35.598260+0000","last_peered":"2024-08-19T01:31:35.598260+0000","last_clean":"2024-08-19T01:31:35.598260+0000","last_became_active":"2024-08-19T01:30:49.538953+0000","last_became_peered":"2024-08-19T01:30:49.538953+0000","last_unstale":"2024-08-19T01:31:35.598260+0000","last_undegraded":"2024-08-19T01:31:35.598260+0000","last_fullsized":"2024-08-19T01:31:35.598260+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:13:17.845140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.870021+0000","last_change":"2024-08-19T01:31:35.870021+0000","last_active":"2024-08-19T01:31:35.870021+0000","last_peered":"2024-08-19T01:31:35.870021+0000","last_clean":"2024-08-19T01:31:35.870021+0000","last_became_active":"2024-08-19T01:31:35.869742+0000","last_became_peered":"2024-08-19T01:31:35.869742+0000","last_unstale":"2024-08-19T01:31:35.870021+0000","last_undegraded":"2024-08-19T01:31:35.870021+0000","last_fullsized":"2024-08-19T01:31:35.870021+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:51.848694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597559+0000","last_change":"2024-08-19T01:30:58.590385+0000","last_active":"2024-08-19T01:31:35.597559+0000","last_peered":"2024-08-19T01:31:35.597559+0000","last_clean":"2024-08-19T01:31:35.597559+0000","last_became_active":"2024-08-19T01:30:58.590004+0000","last_became_peered":"2024-08-19T01:30:58.590004+0000","last_unstale":"2024-08-19T01:31:35.597559+0000","last_undegraded":"2024-08-19T01:31:35.597559+0000","last_fullsized":"2024-08-19T01:31:35.597559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:13:01.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:46.623494+0000","last_change":"2024-08-19T01:31:35.614962+0000","last_active":"2024-08-19T01:31:46.623494+0000","last_peered":"2024-08-19T01:31:46.623494+0000","last_clean":"2024-08-19T01:31:46.623494+0000","last_became_active":"2024-08-19T01:31:35.605904+0000","last_became_peered":"2024-08-19T01:31:35.605904+0000","last_unstale":"2024-08-19T01:31:46.623494+0000","last_undegraded":"2024-08-19T01:31:46.623494+0000","last_fullsized":"2024-08-19T01:31:46.623494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:26:36.102805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603484+0000","last_change":"2024-08-19T01:30:51.550448+0000","last_active":"2024-08-19T01:31:35.603484+0000","last_peered":"2024-08-19T01:31:35.603484+0000","last_clean":"2024-08-19T01:31:35.603484+0000","last_became_active":"2024-08-19T01:30:51.550074+0000","last_became_peered":"2024-08-19T01:30:51.550074+0000","last_unstale":"2024-08-19T01:31:35.603484+0000","last_undegraded":"2024-08-19T01:31:35.603484+0000","last_fullsized":"2024-08-19T01:31:35.603484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:33:13.016183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597794+0000","last_change":"2024-08-19T01:30:58.592077+0000","last_active":"2024-08-19T01:31:35.597794+0000","last_peered":"2024-08-19T01:31:35.597794+0000","last_clean":"2024-08-19T01:31:35.597794+0000","last_became_active":"2024-08-19T01:30:58.591651+0000","last_became_peered":"2024-08-19T01:30:58.591651+0000","last_unstale":"2024-08-19T01:31:35.597794+0000","last_undegraded":"2024-08-19T01:31:35.597794+0000","last_fullsized":"2024-08-19T01:31:35.597794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:09:24.980954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599538+0000","last_change":"2024-08-19T01:30:51.552772+0000","last_active":"2024-08-19T01:31:35.599538+0000","last_peered":"2024-08-19T01:31:35.599538+0000","last_clean":"2024-08-19T01:31:35.599538+0000","last_became_active":"2024-08-19T01:30:51.550904+0000","last_became_peered":"2024-08-19T01:30:51.550904+0000","last_unstale":"2024-08-19T01:31:35.599538+0000","last_undegraded":"2024-08-19T01:31:35.599538+0000","last_fullsized":"2024-08-19T01:31:35.599538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:21:50.082587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598023+0000","last_change":"2024-08-19T01:30:51.559756+0000","last_active":"2024-08-19T01:31:35.598023+0000","last_peered":"2024-08-19T01:31:35.598023+0000","last_clean":"2024-08-19T01:31:35.598023+0000","last_became_active":"2024-08-19T01:30:51.559370+0000","last_became_peered":"2024-08-19T01:30:51.559370+0000","last_unstale":"2024-08-19T01:31:35.598023+0000","last_undegraded":"2024-08-19T01:31:35.598023+0000","last_fullsized":"2024-08-19T01:31:35.598023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:27:30.525346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597386+0000","last_change":"2024-08-19T01:30:51.556577+0000","last_active":"2024-08-19T01:31:35.597386+0000","last_peered":"2024-08-19T01:31:35.597386+0000","last_clean":"2024-08-19T01:31:35.597386+0000","last_became_active":"2024-08-19T01:30:51.555929+0000","last_became_peered":"2024-08-19T01:30:51.555929+0000","last_unstale":"2024-08-19T01:31:35.597386+0000","last_undegraded":"2024-08-19T01:31:35.597386+0000","last_fullsized":"2024-08-19T01:31:35.597386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:46:31.660752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598037+0000","last_change":"2024-08-19T01:30:51.558626+0000","last_active":"2024-08-19T01:31:35.598037+0000","last_peered":"2024-08-19T01:31:35.598037+0000","last_clean":"2024-08-19T01:31:35.598037+0000","last_became_active":"2024-08-19T01:30:51.558188+0000","last_became_peered":"2024-08-19T01:30:51.558188+0000","last_unstale":"2024-08-19T01:31:35.598037+0000","last_undegraded":"2024-08-19T01:31:35.598037+0000","last_fullsized":"2024-08-19T01:31:35.598037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:43:44.777643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745491+0000","last_change":"2024-08-19T01:30:51.552870+0000","last_active":"2024-08-19T01:31:35.745491+0000","last_peered":"2024-08-19T01:31:35.745491+0000","last_clean":"2024-08-19T01:31:35.745491+0000","last_became_active":"2024-08-19T01:30:51.552514+0000","last_became_peered":"2024-08-19T01:30:51.552514+0000","last_unstale":"2024-08-19T01:31:35.745491+0000","last_undegraded":"2024-08-19T01:31:35.745491+0000","last_fullsized":"2024-08-19T01:31:35.745491+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:10:58.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600636+0000","last_change":"2024-08-19T01:30:51.554216+0000","last_active":"2024-08-19T01:31:35.600636+0000","last_peered":"2024-08-19T01:31:35.600636+0000","last_clean":"2024-08-19T01:31:35.600636+0000","last_became_active":"2024-08-19T01:30:51.552535+0000","last_became_peered":"2024-08-19T01:30:51.552535+0000","last_unstale":"2024-08-19T01:31:35.600636+0000","last_undegraded":"2024-08-19T01:31:35.600636+0000","last_fullsized":"2024-08-19T01:31:35.600636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:26:52.971897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598164+0000","last_change":"2024-08-19T01:30:51.561703+0000","last_active":"2024-08-19T01:31:35.598164+0000","last_peered":"2024-08-19T01:31:35.598164+0000","last_clean":"2024-08-19T01:31:35.598164+0000","last_became_active":"2024-08-19T01:30:51.561385+0000","last_became_peered":"2024-08-19T01:30:51.561385+0000","last_unstale":"2024-08-19T01:31:35.598164+0000","last_undegraded":"2024-08-19T01:31:35.598164+0000","last_fullsized":"2024-08-19T01:31:35.598164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:07:23.187023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868940+0000","last_change":"2024-08-19T01:30:51.564047+0000","last_active":"2024-08-19T01:31:35.868940+0000","last_peered":"2024-08-19T01:31:35.868940+0000","last_clean":"2024-08-19T01:31:35.868940+0000","last_became_active":"2024-08-19T01:30:51.563743+0000","last_became_peered":"2024-08-19T01:30:51.563743+0000","last_unstale":"2024-08-19T01:31:35.868940+0000","last_undegraded":"2024-08-19T01:31:35.868940+0000","last_fullsized":"2024-08-19T01:31:35.868940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:50:37.639496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601317+0000","last_change":"2024-08-19T01:30:51.557058+0000","last_active":"2024-08-19T01:31:35.601317+0000","last_peered":"2024-08-19T01:31:35.601317+0000","last_clean":"2024-08-19T01:31:35.601317+0000","last_became_active":"2024-08-19T01:30:51.554611+0000","last_became_peered":"2024-08-19T01:30:51.554611+0000","last_unstale":"2024-08-19T01:31:35.601317+0000","last_undegraded":"2024-08-19T01:31:35.601317+0000","last_fullsized":"2024-08-19T01:31:35.601317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:56:25.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868614+0000","last_change":"2024-08-19T01:30:51.564119+0000","last_active":"2024-08-19T01:31:35.868614+0000","last_peered":"2024-08-19T01:31:35.868614+0000","last_clean":"2024-08-19T01:31:35.868614+0000","last_became_active":"2024-08-19T01:30:51.563840+0000","last_became_peered":"2024-08-19T01:30:51.563840+0000","last_unstale":"2024-08-19T01:31:35.868614+0000","last_undegraded":"2024-08-19T01:31:35.868614+0000","last_fullsized":"2024-08-19T01:31:35.868614+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:44:44.928760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596119+0000","last_change":"2024-08-19T01:30:51.561639+0000","last_active":"2024-08-19T01:31:35.596119+0000","last_peered":"2024-08-19T01:31:35.596119+0000","last_clean":"2024-08-19T01:31:35.596119+0000","last_became_active":"2024-08-19T01:30:51.561207+0000","last_became_peered":"2024-08-19T01:30:51.561207+0000","last_unstale":"2024-08-19T01:31:35.596119+0000","last_undegraded":"2024-08-19T01:31:35.596119+0000","last_fullsized":"2024-08-19T01:31:35.596119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:12:20.076628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745343+0000","last_change":"2024-08-19T01:30:51.556700+0000","last_active":"2024-08-19T01:31:35.745343+0000","last_peered":"2024-08-19T01:31:35.745343+0000","last_clean":"2024-08-19T01:31:35.745343+0000","last_became_active":"2024-08-19T01:30:51.556407+0000","last_became_peered":"2024-08-19T01:30:51.556407+0000","last_unstale":"2024-08-19T01:31:35.745343+0000","last_undegraded":"2024-08-19T01:31:35.745343+0000","last_fullsized":"2024-08-19T01:31:35.745343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:03:16.664253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599693+0000","last_change":"2024-08-19T01:30:51.553764+0000","last_active":"2024-08-19T01:31:35.599693+0000","last_peered":"2024-08-19T01:31:35.599693+0000","last_clean":"2024-08-19T01:31:35.599693+0000","last_became_active":"2024-08-19T01:30:51.553442+0000","last_became_peered":"2024-08-19T01:30:51.553442+0000","last_unstale":"2024-08-19T01:31:35.599693+0000","last_undegraded":"2024-08-19T01:31:35.599693+0000","last_fullsized":"2024-08-19T01:31:35.599693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:36:44.476979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598004+0000","last_change":"2024-08-19T01:30:51.565528+0000","last_active":"2024-08-19T01:31:35.598004+0000","last_peered":"2024-08-19T01:31:35.598004+0000","last_clean":"2024-08-19T01:31:35.598004+0000","last_became_active":"2024-08-19T01:30:51.564649+0000","last_became_peered":"2024-08-19T01:30:51.564649+0000","last_unstale":"2024-08-19T01:31:35.598004+0000","last_undegraded":"2024-08-19T01:31:35.598004+0000","last_fullsized":"2024-08-19T01:31:35.598004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:04:12.868496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595505+0000","last_change":"2024-08-19T01:30:51.553425+0000","last_active":"2024-08-19T01:31:35.595505+0000","last_peered":"2024-08-19T01:31:35.595505+0000","last_clean":"2024-08-19T01:31:35.595505+0000","last_became_active":"2024-08-19T01:30:51.553123+0000","last_became_peered":"2024-08-19T01:30:51.553123+0000","last_unstale":"2024-08-19T01:31:35.595505+0000","last_undegraded":"2024-08-19T01:31:35.595505+0000","last_fullsized":"2024-08-19T01:31:35.595505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:07.969228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600659+0000","last_change":"2024-08-19T01:30:51.556820+0000","last_active":"2024-08-19T01:31:35.600659+0000","last_peered":"2024-08-19T01:31:35.600659+0000","last_clean":"2024-08-19T01:31:35.600659+0000","last_became_active":"2024-08-19T01:30:51.556532+0000","last_became_peered":"2024-08-19T01:30:51.556532+0000","last_unstale":"2024-08-19T01:31:35.600659+0000","last_undegraded":"2024-08-19T01:31:35.600659+0000","last_fullsized":"2024-08-19T01:31:35.600659+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:40:37.053193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595799+0000","last_change":"2024-08-19T01:30:51.561610+0000","last_active":"2024-08-19T01:31:35.595799+0000","last_peered":"2024-08-19T01:31:35.595799+0000","last_clean":"2024-08-19T01:31:35.595799+0000","last_became_active":"2024-08-19T01:30:51.561270+0000","last_became_peered":"2024-08-19T01:30:51.561270+0000","last_unstale":"2024-08-19T01:31:35.595799+0000","last_undegraded":"2024-08-19T01:31:35.595799+0000","last_fullsized":"2024-08-19T01:31:35.595799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:15:15.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597839+0000","last_change":"2024-08-19T01:30:51.558128+0000","last_active":"2024-08-19T01:31:35.597839+0000","last_peered":"2024-08-19T01:31:35.597839+0000","last_clean":"2024-08-19T01:31:35.597839+0000","last_became_active":"2024-08-19T01:30:51.557785+0000","last_became_peered":"2024-08-19T01:30:51.557785+0000","last_unstale":"2024-08-19T01:31:35.597839+0000","last_undegraded":"2024-08-19T01:31:35.597839+0000","last_fullsized":"2024-08-19T01:31:35.597839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:14.849212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597900+0000","last_change":"2024-08-19T01:30:51.553917+0000","last_active":"2024-08-19T01:31:35.597900+0000","last_peered":"2024-08-19T01:31:35.597900+0000","last_clean":"2024-08-19T01:31:35.597900+0000","last_became_active":"2024-08-19T01:30:51.553596+0000","last_became_peered":"2024-08-19T01:30:51.553596+0000","last_unstale":"2024-08-19T01:31:35.597900+0000","last_undegraded":"2024-08-19T01:31:35.597900+0000","last_fullsized":"2024-08-19T01:31:35.597900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:03:36.001486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745336+0000","last_change":"2024-08-19T01:30:51.555201+0000","last_active":"2024-08-19T01:31:35.745336+0000","last_peered":"2024-08-19T01:31:35.745336+0000","last_clean":"2024-08-19T01:31:35.745336+0000","last_became_active":"2024-08-19T01:30:51.554160+0000","last_became_peered":"2024-08-19T01:30:51.554160+0000","last_unstale":"2024-08-19T01:31:35.745336+0000","last_undegraded":"2024-08-19T01:31:35.745336+0000","last_fullsized":"2024-08-19T01:31:35.745336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:58:20.586931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600619+0000","last_change":"2024-08-19T01:30:51.560517+0000","last_active":"2024-08-19T01:31:35.600619+0000","last_peered":"2024-08-19T01:31:35.600619+0000","last_clean":"2024-08-19T01:31:35.600619+0000","last_became_active":"2024-08-19T01:30:51.557454+0000","last_became_peered":"2024-08-19T01:30:51.557454+0000","last_unstale":"2024-08-19T01:31:35.600619+0000","last_undegraded":"2024-08-19T01:31:35.600619+0000","last_fullsized":"2024-08-19T01:31:35.600619+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:41:08.192165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:51.565961+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:51.565491+0000","last_became_peered":"2024-08-19T01:30:51.565491+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:53:06.263411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599327+0000","last_change":"2024-08-19T01:30:51.551297+0000","last_active":"2024-08-19T01:31:35.599327+0000","last_peered":"2024-08-19T01:31:35.599327+0000","last_clean":"2024-08-19T01:31:35.599327+0000","last_became_active":"2024-08-19T01:30:51.550929+0000","last_became_peered":"2024-08-19T01:30:51.550929+0000","last_unstale":"2024-08-19T01:31:35.599327+0000","last_undegraded":"2024-08-19T01:31:35.599327+0000","last_fullsized":"2024-08-19T01:31:35.599327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:07:03.625360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603232+0000","last_change":"2024-08-19T01:31:35.602883+0000","last_active":"2024-08-19T01:31:35.603232+0000","last_peered":"2024-08-19T01:31:35.603232+0000","last_clean":"2024-08-19T01:31:35.603232+0000","last_became_active":"2024-08-19T01:31:35.602571+0000","last_became_peered":"2024-08-19T01:31:35.602571+0000","last_unstale":"2024-08-19T01:31:35.603232+0000","last_undegraded":"2024-08-19T01:31:35.603232+0000","last_fullsized":"2024-08-19T01:31:35.603232+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:08:13.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868405+0000","last_change":"2024-08-19T01:30:51.552548+0000","last_active":"2024-08-19T01:31:35.868405+0000","last_peered":"2024-08-19T01:31:35.868405+0000","last_clean":"2024-08-19T01:31:35.868405+0000","last_became_active":"2024-08-19T01:30:51.552264+0000","last_became_peered":"2024-08-19T01:30:51.552264+0000","last_unstale":"2024-08-19T01:31:35.868405+0000","last_undegraded":"2024-08-19T01:31:35.868405+0000","last_fullsized":"2024-08-19T01:31:35.868405+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:17:53.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603728+0000","last_change":"2024-08-19T01:30:51.558686+0000","last_active":"2024-08-19T01:31:35.603728+0000","last_peered":"2024-08-19T01:31:35.603728+0000","last_clean":"2024-08-19T01:31:35.603728+0000","last_became_active":"2024-08-19T01:30:51.558190+0000","last_became_peered":"2024-08-19T01:30:51.558190+0000","last_unstale":"2024-08-19T01:31:35.603728+0000","last_undegraded":"2024-08-19T01:31:35.603728+0000","last_fullsized":"2024-08-19T01:31:35.603728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:52:33.056948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598221+0000","last_change":"2024-08-19T01:30:51.548868+0000","last_active":"2024-08-19T01:31:35.598221+0000","last_peered":"2024-08-19T01:31:35.598221+0000","last_clean":"2024-08-19T01:31:35.598221+0000","last_became_active":"2024-08-19T01:30:51.548347+0000","last_became_peered":"2024-08-19T01:30:51.548347+0000","last_unstale":"2024-08-19T01:31:35.598221+0000","last_undegraded":"2024-08-19T01:31:35.598221+0000","last_fullsized":"2024-08-19T01:31:35.598221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:39:22.049290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603369+0000","last_change":"2024-08-19T01:30:51.554054+0000","last_active":"2024-08-19T01:31:35.603369+0000","last_peered":"2024-08-19T01:31:35.603369+0000","last_clean":"2024-08-19T01:31:35.603369+0000","last_became_active":"2024-08-19T01:30:51.553731+0000","last_became_peered":"2024-08-19T01:30:51.553731+0000","last_unstale":"2024-08-19T01:31:35.603369+0000","last_undegraded":"2024-08-19T01:31:35.603369+0000","last_fullsized":"2024-08-19T01:31:35.603369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:25:17.939024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600261+0000","last_change":"2024-08-19T01:30:51.558662+0000","last_active":"2024-08-19T01:31:35.600261+0000","last_peered":"2024-08-19T01:31:35.600261+0000","last_clean":"2024-08-19T01:31:35.600261+0000","last_became_active":"2024-08-19T01:30:51.558131+0000","last_became_peered":"2024-08-19T01:30:51.558131+0000","last_unstale":"2024-08-19T01:31:35.600261+0000","last_undegraded":"2024-08-19T01:31:35.600261+0000","last_fullsized":"2024-08-19T01:31:35.600261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:31:25.054005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598117+0000","last_change":"2024-08-19T01:30:51.565170+0000","last_active":"2024-08-19T01:31:35.598117+0000","last_peered":"2024-08-19T01:31:35.598117+0000","last_clean":"2024-08-19T01:31:35.598117+0000","last_became_active":"2024-08-19T01:30:51.564765+0000","last_became_peered":"2024-08-19T01:30:51.564765+0000","last_unstale":"2024-08-19T01:31:35.598117+0000","last_undegraded":"2024-08-19T01:31:35.598117+0000","last_fullsized":"2024-08-19T01:31:35.598117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:00:48.668865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598106+0000","last_change":"2024-08-19T01:30:51.555300+0000","last_active":"2024-08-19T01:31:35.598106+0000","last_peered":"2024-08-19T01:31:35.598106+0000","last_clean":"2024-08-19T01:31:35.598106+0000","last_became_active":"2024-08-19T01:30:51.554977+0000","last_became_peered":"2024-08-19T01:30:51.554977+0000","last_unstale":"2024-08-19T01:31:35.598106+0000","last_undegraded":"2024-08-19T01:31:35.598106+0000","last_fullsized":"2024-08-19T01:31:35.598106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:18:58.961783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600410+0000","last_change":"2024-08-19T01:30:51.557906+0000","last_active":"2024-08-19T01:31:35.600410+0000","last_peered":"2024-08-19T01:31:35.600410+0000","last_clean":"2024-08-19T01:31:35.600410+0000","last_became_active":"2024-08-19T01:30:51.557598+0000","last_became_peered":"2024-08-19T01:30:51.557598+0000","last_unstale":"2024-08-19T01:31:35.600410+0000","last_undegraded":"2024-08-19T01:31:35.600410+0000","last_fullsized":"2024-08-19T01:31:35.600410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:33:12.785910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61057,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":100,"num_read_kb":110,"num_write":158,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4463,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":156515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282761,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481815,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":513466,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1157791,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":517798,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168612,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":513622,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430250,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1109920,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593775,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":508878,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822708,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1105118,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986232,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":557257,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215163,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":509072,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":509516,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":553139,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":510266,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":44351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":44373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T01:31:50.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T01:31:50.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:50 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:50.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:50 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:51.068 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:51.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 -- 172.21.15.64:0/2275192505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 msgr2=0x7ff0e810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:51.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 --2- 172.21.15.64:0/2275192505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e810e7b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009cd0 tx=0x7ff0e002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 -- 172.21.15.64:0/2275192505 shutdown_connections 2024-08-19T01:31:51.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 --2- 172.21.15.64:0/2275192505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e810e7b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 --2- 172.21.15.64:0/2275192505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0e81018e0 0x7ff0e8101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 --2- 172.21.15.64:0/2275192505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0e81078e0 0x7ff0e8107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 -- 172.21.15.64:0/2275192505 >> 172.21.15.64:0/2275192505 conn(0x7ff0e80fd600 msgr2=0x7ff0e80ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 -- 172.21.15.64:0/2275192505 shutdown_connections 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 -- 172.21.15.64:0/2275192505 wait complete. 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 Processor -- start 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.545+0000 7ff0eca70640 1 -- start start 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0eca70640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0e81018e0 0x7ff0e81981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0eca70640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0eca70640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0e81078e0 0x7ff0e819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0eca70640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8111ce0 con 0x7ff0e8102280 2024-08-19T01:31:51.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0eca70640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8111b60 con 0x7ff0e81078e0 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0eca70640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8111e60 con 0x7ff0e81018e0 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e81986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38408/0 (socket says 172.21.15.64:38408) 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 -- 172.21.15.64:0/1088801045 learned_addr learned my addr 172.21.15.64:0/1088801045 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e6d9d640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0e81078e0 0x7ff0e819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e659c640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0e81018e0 0x7ff0e81981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 -- 172.21.15.64:0/1088801045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0e81018e0 msgr2=0x7ff0e81981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0e81018e0 0x7ff0e81981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 -- 172.21.15.64:0/1088801045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0e81078e0 msgr2=0x7ff0e819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0e81078e0 0x7ff0e819ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 -- 172.21.15.64:0/1088801045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e819d150 con 0x7ff0e8102280 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e659c640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0e81018e0 0x7ff0e81981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0e5d9b640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e81986e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000d770 tx=0x7ff0d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.546+0000 7ff0df7fe640 1 -- 172.21.15.64:0/1088801045 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0017070 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.547+0000 7ff0df7fe640 1 -- 172.21.15.64:0/1088801045 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d0005500 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.547+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e819d440 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.547+0000 7ff0df7fe640 1 -- 172.21.15.64:0/1088801045 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0004e00 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.547+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e81a4d20 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.548+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e8101d40 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.549+0000 7ff0df7fe640 1 -- 172.21.15.64:0/1088801045 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff0d0004fa0 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.550+0000 7ff0df7fe640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0d40a2c90 0x7ff0d40a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.550+0000 7ff0e659c640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0d40a2c90 0x7ff0d40a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.550+0000 7ff0df7fe640 1 -- 172.21.15.64:0/1088801045 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff0d00cf370 con 0x7ff0e8102280 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.550+0000 7ff0e659c640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0d40a2c90 0x7ff0d40a5150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8005fd0 tx=0x7ff0d8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.553+0000 7ff0df7fe640 1 -- 172.21.15.64:0/1088801045 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff0d009be90 con 0x7ff0e8102280 2024-08-19T01:31:51.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.727+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff0e8062360 con 0x7ff0d40a2c90 2024-08-19T01:31:51.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:51 smithi064 ceph-mon[18515]: pgmap v299: 129 pgs: 129 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.6 KiB/s wr, 9 op/s; 371 B/s, 0 objects/s recovering 2024-08-19T01:31:51.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:51 smithi064 ceph-mon[18515]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:31:51.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:31:51.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:31:51.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:51.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.735+0000 7ff0df7fe640 1 -- 172.21.15.64:0/1088801045 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340638 (secure 0 0 0) 0x7ff0e8062360 con 0x7ff0d40a2c90 2024-08-19T01:31:51.742 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:51.744 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0d40a2c90 msgr2=0x7ff0d40a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0d40a2c90 0x7ff0d40a5150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8005fd0 tx=0x7ff0d8005850 comp rx=0 tx=0).stop 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 msgr2=0x7ff0e81986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e81986e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000d770 tx=0x7ff0d000dc30 comp rx=0 tx=0).stop 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 shutdown_connections 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0d40a2c90 0x7ff0d40a5150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0e81078e0 0x7ff0e819ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0e8102280 0x7ff0e81986e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 --2- 172.21.15.64:0/1088801045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0e81018e0 0x7ff0e81981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:51.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 >> 172.21.15.64:0/1088801045 conn(0x7ff0e80fd600 msgr2=0x7ff0e80fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:51.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 shutdown_connections 2024-08-19T01:31:51.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:51.739+0000 7ff0eca70640 1 -- 172.21.15.64:0/1088801045 wait complete. 2024-08-19T01:31:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:51 smithi123 ceph-mon[21829]: pgmap v299: 129 pgs: 129 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.6 KiB/s wr, 9 op/s; 371 B/s, 0 objects/s recovering 2024-08-19T01:31:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:51 smithi123 ceph-mon[21829]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:31:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:31:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:31:51.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:51 smithi161 ceph-mon[21757]: pgmap v299: 129 pgs: 129 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.6 KiB/s wr, 9 op/s; 371 B/s, 0 objects/s recovering 2024-08-19T01:31:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:51 smithi161 ceph-mon[21757]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:31:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:31:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:31:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:31:52.514 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-08-19T01:31:51.687091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651659,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":246,"num_read_kb":234,"num_write":348,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1185199,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361412,"kb_used_data":12236,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580412,"statfs":{"total":1151940427776,"available":1151570341888,"internally_reserved":0,"allocated":12529664,"data_stored":8092499,"data_compressed":390537,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46851,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":83,"num_write":33,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595389+0000","last_change":"2024-08-19T01:30:49.539512+0000","last_active":"2024-08-19T01:31:35.595389+0000","last_peered":"2024-08-19T01:31:35.595389+0000","last_clean":"2024-08-19T01:31:35.595389+0000","last_became_active":"2024-08-19T01:30:49.539163+0000","last_became_peered":"2024-08-19T01:30:49.539163+0000","last_unstale":"2024-08-19T01:31:35.595389+0000","last_undegraded":"2024-08-19T01:31:35.595389+0000","last_fullsized":"2024-08-19T01:31:35.595389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:23.874916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868682+0000","last_change":"2024-08-19T01:30:51.561135+0000","last_active":"2024-08-19T01:31:35.868682+0000","last_peered":"2024-08-19T01:31:35.868682+0000","last_clean":"2024-08-19T01:31:35.868682+0000","last_became_active":"2024-08-19T01:30:51.560822+0000","last_became_peered":"2024-08-19T01:30:51.560822+0000","last_unstale":"2024-08-19T01:31:35.868682+0000","last_undegraded":"2024-08-19T01:31:35.868682+0000","last_fullsized":"2024-08-19T01:31:35.868682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:08:35.918895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597319+0000","last_change":"2024-08-19T01:30:58.595727+0000","last_active":"2024-08-19T01:31:35.597319+0000","last_peered":"2024-08-19T01:31:35.597319+0000","last_clean":"2024-08-19T01:31:35.597319+0000","last_became_active":"2024-08-19T01:30:58.595286+0000","last_became_peered":"2024-08-19T01:30:58.595286+0000","last_unstale":"2024-08-19T01:31:35.597319+0000","last_undegraded":"2024-08-19T01:31:35.597319+0000","last_fullsized":"2024-08-19T01:31:35.597319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:25:02.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868994+0000","last_change":"2024-08-19T01:30:49.575693+0000","last_active":"2024-08-19T01:31:35.868994+0000","last_peered":"2024-08-19T01:31:35.868994+0000","last_clean":"2024-08-19T01:31:35.868994+0000","last_became_active":"2024-08-19T01:30:49.575331+0000","last_became_peered":"2024-08-19T01:30:49.575331+0000","last_unstale":"2024-08-19T01:31:35.868994+0000","last_undegraded":"2024-08-19T01:31:35.868994+0000","last_fullsized":"2024-08-19T01:31:35.868994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:31:50.773160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598318+0000","last_change":"2024-08-19T01:30:51.562446+0000","last_active":"2024-08-19T01:31:35.598318+0000","last_peered":"2024-08-19T01:31:35.598318+0000","last_clean":"2024-08-19T01:31:35.598318+0000","last_became_active":"2024-08-19T01:30:51.561475+0000","last_became_peered":"2024-08-19T01:30:51.561475+0000","last_unstale":"2024-08-19T01:31:35.598318+0000","last_undegraded":"2024-08-19T01:31:35.598318+0000","last_fullsized":"2024-08-19T01:31:35.598318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:51:41.531247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599871+0000","last_change":"2024-08-19T01:30:58.591371+0000","last_active":"2024-08-19T01:31:35.599871+0000","last_peered":"2024-08-19T01:31:35.599871+0000","last_clean":"2024-08-19T01:31:35.599871+0000","last_became_active":"2024-08-19T01:30:58.590936+0000","last_became_peered":"2024-08-19T01:30:58.590936+0000","last_unstale":"2024-08-19T01:31:35.599871+0000","last_undegraded":"2024-08-19T01:31:35.599871+0000","last_fullsized":"2024-08-19T01:31:35.599871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:07:33.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.613588+0000","last_change":"2024-08-19T01:31:35.613145+0000","last_active":"2024-08-19T01:31:35.613588+0000","last_peered":"2024-08-19T01:31:35.613588+0000","last_clean":"2024-08-19T01:31:35.613588+0000","last_became_active":"2024-08-19T01:31:35.605087+0000","last_became_peered":"2024-08-19T01:31:35.605087+0000","last_unstale":"2024-08-19T01:31:35.613588+0000","last_undegraded":"2024-08-19T01:31:35.613588+0000","last_fullsized":"2024-08-19T01:31:35.613588+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:25:59.109707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596225+0000","last_change":"2024-08-19T01:30:51.555256+0000","last_active":"2024-08-19T01:31:35.596225+0000","last_peered":"2024-08-19T01:31:35.596225+0000","last_clean":"2024-08-19T01:31:35.596225+0000","last_became_active":"2024-08-19T01:30:51.554030+0000","last_became_peered":"2024-08-19T01:30:51.554030+0000","last_unstale":"2024-08-19T01:31:35.596225+0000","last_undegraded":"2024-08-19T01:31:35.596225+0000","last_fullsized":"2024-08-19T01:31:35.596225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:12:04.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600498+0000","last_change":"2024-08-19T01:30:58.589362+0000","last_active":"2024-08-19T01:31:35.600498+0000","last_peered":"2024-08-19T01:31:35.600498+0000","last_clean":"2024-08-19T01:31:35.600498+0000","last_became_active":"2024-08-19T01:30:58.588937+0000","last_became_peered":"2024-08-19T01:30:58.588937+0000","last_unstale":"2024-08-19T01:31:35.600498+0000","last_undegraded":"2024-08-19T01:31:35.600498+0000","last_fullsized":"2024-08-19T01:31:35.600498+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:31.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604267+0000","last_change":"2024-08-19T01:30:49.538597+0000","last_active":"2024-08-19T01:31:35.604267+0000","last_peered":"2024-08-19T01:31:35.604267+0000","last_clean":"2024-08-19T01:31:35.604267+0000","last_became_active":"2024-08-19T01:30:49.538228+0000","last_became_peered":"2024-08-19T01:30:49.538228+0000","last_unstale":"2024-08-19T01:31:35.604267+0000","last_undegraded":"2024-08-19T01:31:35.604267+0000","last_fullsized":"2024-08-19T01:31:35.604267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:31.350962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600891+0000","last_change":"2024-08-19T01:30:51.562829+0000","last_active":"2024-08-19T01:31:35.600891+0000","last_peered":"2024-08-19T01:31:35.600891+0000","last_clean":"2024-08-19T01:31:35.600891+0000","last_became_active":"2024-08-19T01:30:51.562144+0000","last_became_peered":"2024-08-19T01:30:51.562144+0000","last_unstale":"2024-08-19T01:31:35.600891+0000","last_undegraded":"2024-08-19T01:31:35.600891+0000","last_fullsized":"2024-08-19T01:31:35.600891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:17:09.184222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599645+0000","last_change":"2024-08-19T01:30:58.593178+0000","last_active":"2024-08-19T01:31:35.599645+0000","last_peered":"2024-08-19T01:31:35.599645+0000","last_clean":"2024-08-19T01:31:35.599645+0000","last_became_active":"2024-08-19T01:30:58.591247+0000","last_became_peered":"2024-08-19T01:30:58.591247+0000","last_unstale":"2024-08-19T01:31:35.599645+0000","last_undegraded":"2024-08-19T01:31:35.599645+0000","last_fullsized":"2024-08-19T01:31:35.599645+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:20:40.796135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604428+0000","last_change":"2024-08-19T01:30:51.562307+0000","last_active":"2024-08-19T01:31:35.604428+0000","last_peered":"2024-08-19T01:31:35.604428+0000","last_clean":"2024-08-19T01:31:35.604428+0000","last_became_active":"2024-08-19T01:30:51.561957+0000","last_became_peered":"2024-08-19T01:30:51.561957+0000","last_unstale":"2024-08-19T01:31:35.604428+0000","last_undegraded":"2024-08-19T01:31:35.604428+0000","last_fullsized":"2024-08-19T01:31:35.604428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:22.902317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.610093+0000","last_change":"2024-08-19T01:31:35.609612+0000","last_active":"2024-08-19T01:31:35.610093+0000","last_peered":"2024-08-19T01:31:35.610093+0000","last_clean":"2024-08-19T01:31:35.610093+0000","last_became_active":"2024-08-19T01:31:35.604969+0000","last_became_peered":"2024-08-19T01:31:35.604969+0000","last_unstale":"2024-08-19T01:31:35.610093+0000","last_undegraded":"2024-08-19T01:31:35.610093+0000","last_fullsized":"2024-08-19T01:31:35.610093+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:59:38.089119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600108+0000","last_change":"2024-08-19T01:30:58.588984+0000","last_active":"2024-08-19T01:31:35.600108+0000","last_peered":"2024-08-19T01:31:35.600108+0000","last_clean":"2024-08-19T01:31:35.600108+0000","last_became_active":"2024-08-19T01:30:58.588504+0000","last_became_peered":"2024-08-19T01:30:58.588504+0000","last_unstale":"2024-08-19T01:31:35.600108+0000","last_undegraded":"2024-08-19T01:31:35.600108+0000","last_fullsized":"2024-08-19T01:31:35.600108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:29:14.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869072+0000","last_change":"2024-08-19T01:30:51.553293+0000","last_active":"2024-08-19T01:31:35.869072+0000","last_peered":"2024-08-19T01:31:35.869072+0000","last_clean":"2024-08-19T01:31:35.869072+0000","last_became_active":"2024-08-19T01:30:51.552990+0000","last_became_peered":"2024-08-19T01:30:51.552990+0000","last_unstale":"2024-08-19T01:31:35.869072+0000","last_undegraded":"2024-08-19T01:31:35.869072+0000","last_fullsized":"2024-08-19T01:31:35.869072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:28:33.193014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597523+0000","last_change":"2024-08-19T01:30:49.537742+0000","last_active":"2024-08-19T01:31:35.597523+0000","last_peered":"2024-08-19T01:31:35.597523+0000","last_clean":"2024-08-19T01:31:35.597523+0000","last_became_active":"2024-08-19T01:30:49.537153+0000","last_became_peered":"2024-08-19T01:30:49.537153+0000","last_unstale":"2024-08-19T01:31:35.597523+0000","last_undegraded":"2024-08-19T01:31:35.597523+0000","last_fullsized":"2024-08-19T01:31:35.597523+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:25:03.650900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600493+0000","last_change":"2024-08-19T01:30:58.592348+0000","last_active":"2024-08-19T01:31:35.600493+0000","last_peered":"2024-08-19T01:31:35.600493+0000","last_clean":"2024-08-19T01:31:35.600493+0000","last_became_active":"2024-08-19T01:30:58.591641+0000","last_became_peered":"2024-08-19T01:30:58.591641+0000","last_unstale":"2024-08-19T01:31:35.600493+0000","last_undegraded":"2024-08-19T01:31:35.600493+0000","last_fullsized":"2024-08-19T01:31:35.600493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:39:17.542800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599683+0000","last_change":"2024-08-19T01:30:51.552126+0000","last_active":"2024-08-19T01:31:35.599683+0000","last_peered":"2024-08-19T01:31:35.599683+0000","last_clean":"2024-08-19T01:31:35.599683+0000","last_became_active":"2024-08-19T01:30:51.551795+0000","last_became_peered":"2024-08-19T01:30:51.551795+0000","last_unstale":"2024-08-19T01:31:35.599683+0000","last_undegraded":"2024-08-19T01:31:35.599683+0000","last_fullsized":"2024-08-19T01:31:35.599683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:22:54.705359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868723+0000","last_change":"2024-08-19T01:30:49.536659+0000","last_active":"2024-08-19T01:31:35.868723+0000","last_peered":"2024-08-19T01:31:35.868723+0000","last_clean":"2024-08-19T01:31:35.868723+0000","last_became_active":"2024-08-19T01:30:49.536366+0000","last_became_peered":"2024-08-19T01:30:49.536366+0000","last_unstale":"2024-08-19T01:31:35.868723+0000","last_undegraded":"2024-08-19T01:31:35.868723+0000","last_fullsized":"2024-08-19T01:31:35.868723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:58:52.547461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595100+0000","last_change":"2024-08-19T01:30:58.593724+0000","last_active":"2024-08-19T01:31:35.595100+0000","last_peered":"2024-08-19T01:31:35.595100+0000","last_clean":"2024-08-19T01:31:35.595100+0000","last_became_active":"2024-08-19T01:30:58.593383+0000","last_became_peered":"2024-08-19T01:30:58.593383+0000","last_unstale":"2024-08-19T01:31:35.595100+0000","last_undegraded":"2024-08-19T01:31:35.595100+0000","last_fullsized":"2024-08-19T01:31:35.595100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:54:26.424139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603836+0000","last_change":"2024-08-19T01:30:51.557682+0000","last_active":"2024-08-19T01:31:35.603836+0000","last_peered":"2024-08-19T01:31:35.603836+0000","last_clean":"2024-08-19T01:31:35.603836+0000","last_became_active":"2024-08-19T01:30:51.557351+0000","last_became_peered":"2024-08-19T01:30:51.557351+0000","last_unstale":"2024-08-19T01:31:35.603836+0000","last_undegraded":"2024-08-19T01:31:35.603836+0000","last_fullsized":"2024-08-19T01:31:35.603836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:41.484955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595958+0000","last_change":"2024-08-19T01:30:49.575996+0000","last_active":"2024-08-19T01:31:35.595958+0000","last_peered":"2024-08-19T01:31:35.595958+0000","last_clean":"2024-08-19T01:31:35.595958+0000","last_became_active":"2024-08-19T01:30:49.575607+0000","last_became_peered":"2024-08-19T01:30:49.575607+0000","last_unstale":"2024-08-19T01:31:35.595958+0000","last_undegraded":"2024-08-19T01:31:35.595958+0000","last_fullsized":"2024-08-19T01:31:35.595958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:20:33.467100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600113+0000","last_change":"2024-08-19T01:30:58.592629+0000","last_active":"2024-08-19T01:31:35.600113+0000","last_peered":"2024-08-19T01:31:35.600113+0000","last_clean":"2024-08-19T01:31:35.600113+0000","last_became_active":"2024-08-19T01:30:58.592122+0000","last_became_peered":"2024-08-19T01:30:58.592122+0000","last_unstale":"2024-08-19T01:31:35.600113+0000","last_undegraded":"2024-08-19T01:31:35.600113+0000","last_fullsized":"2024-08-19T01:31:35.600113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:19:09.846829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.746416+0000","last_change":"2024-08-19T01:31:35.746416+0000","last_active":"2024-08-19T01:31:35.746416+0000","last_peered":"2024-08-19T01:31:35.746416+0000","last_clean":"2024-08-19T01:31:35.746416+0000","last_became_active":"2024-08-19T01:31:35.746110+0000","last_became_peered":"2024-08-19T01:31:35.746110+0000","last_unstale":"2024-08-19T01:31:35.746416+0000","last_undegraded":"2024-08-19T01:31:35.746416+0000","last_fullsized":"2024-08-19T01:31:35.746416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:29:29.789785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600943+0000","last_change":"2024-08-19T01:30:49.711366+0000","last_active":"2024-08-19T01:31:35.600943+0000","last_peered":"2024-08-19T01:31:35.600943+0000","last_clean":"2024-08-19T01:31:35.600943+0000","last_became_active":"2024-08-19T01:30:49.711078+0000","last_became_peered":"2024-08-19T01:30:49.711078+0000","last_unstale":"2024-08-19T01:31:35.600943+0000","last_undegraded":"2024-08-19T01:31:35.600943+0000","last_fullsized":"2024-08-19T01:31:35.600943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:11:16.767867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597334+0000","last_change":"2024-08-19T01:30:58.591845+0000","last_active":"2024-08-19T01:31:35.597334+0000","last_peered":"2024-08-19T01:31:35.597334+0000","last_clean":"2024-08-19T01:31:35.597334+0000","last_became_active":"2024-08-19T01:30:58.591453+0000","last_became_peered":"2024-08-19T01:30:58.591453+0000","last_unstale":"2024-08-19T01:31:35.597334+0000","last_undegraded":"2024-08-19T01:31:35.597334+0000","last_fullsized":"2024-08-19T01:31:35.597334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:37:09.521514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869013+0000","last_change":"2024-08-19T01:30:51.563156+0000","last_active":"2024-08-19T01:31:35.869013+0000","last_peered":"2024-08-19T01:31:35.869013+0000","last_clean":"2024-08-19T01:31:35.869013+0000","last_became_active":"2024-08-19T01:30:51.562860+0000","last_became_peered":"2024-08-19T01:30:51.562860+0000","last_unstale":"2024-08-19T01:31:35.869013+0000","last_undegraded":"2024-08-19T01:31:35.869013+0000","last_fullsized":"2024-08-19T01:31:35.869013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:54:11.768833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:48.559439+0000","last_change":"2024-08-19T01:30:49.536648+0000","last_active":"2024-08-19T01:31:48.559439+0000","last_peered":"2024-08-19T01:31:48.559439+0000","last_clean":"2024-08-19T01:31:48.559439+0000","last_became_active":"2024-08-19T01:30:49.536080+0000","last_became_peered":"2024-08-19T01:30:49.536080+0000","last_unstale":"2024-08-19T01:31:48.559439+0000","last_undegraded":"2024-08-19T01:31:48.559439+0000","last_fullsized":"2024-08-19T01:31:48.559439+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:03:46.804252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868291+0000","last_change":"2024-08-19T01:30:58.592253+0000","last_active":"2024-08-19T01:31:35.868291+0000","last_peered":"2024-08-19T01:31:35.868291+0000","last_clean":"2024-08-19T01:31:35.868291+0000","last_became_active":"2024-08-19T01:30:58.591911+0000","last_became_peered":"2024-08-19T01:30:58.591911+0000","last_unstale":"2024-08-19T01:31:35.868291+0000","last_undegraded":"2024-08-19T01:31:35.868291+0000","last_fullsized":"2024-08-19T01:31:35.868291+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:02:27.112597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745492+0000","last_change":"2024-08-19T01:30:51.547607+0000","last_active":"2024-08-19T01:31:35.745492+0000","last_peered":"2024-08-19T01:31:35.745492+0000","last_clean":"2024-08-19T01:31:35.745492+0000","last_became_active":"2024-08-19T01:30:51.547316+0000","last_became_peered":"2024-08-19T01:30:51.547316+0000","last_unstale":"2024-08-19T01:31:35.745492+0000","last_undegraded":"2024-08-19T01:31:35.745492+0000","last_fullsized":"2024-08-19T01:31:35.745492+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:48:58.300508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604727+0000","last_change":"2024-08-19T01:30:49.576942+0000","last_active":"2024-08-19T01:31:35.604727+0000","last_peered":"2024-08-19T01:31:35.604727+0000","last_clean":"2024-08-19T01:31:35.604727+0000","last_became_active":"2024-08-19T01:30:49.576528+0000","last_became_peered":"2024-08-19T01:30:49.576528+0000","last_unstale":"2024-08-19T01:31:35.604727+0000","last_undegraded":"2024-08-19T01:31:35.604727+0000","last_fullsized":"2024-08-19T01:31:35.604727+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:23:53.259395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600274+0000","last_change":"2024-08-19T01:30:58.591577+0000","last_active":"2024-08-19T01:31:35.600274+0000","last_peered":"2024-08-19T01:31:35.600274+0000","last_clean":"2024-08-19T01:31:35.600274+0000","last_became_active":"2024-08-19T01:30:58.591181+0000","last_became_peered":"2024-08-19T01:30:58.591181+0000","last_unstale":"2024-08-19T01:31:35.600274+0000","last_undegraded":"2024-08-19T01:31:35.600274+0000","last_fullsized":"2024-08-19T01:31:35.600274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:02:59.682147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598931+0000","last_change":"2024-08-19T01:30:51.548556+0000","last_active":"2024-08-19T01:31:35.598931+0000","last_peered":"2024-08-19T01:31:35.598931+0000","last_clean":"2024-08-19T01:31:35.598931+0000","last_became_active":"2024-08-19T01:30:51.547838+0000","last_became_peered":"2024-08-19T01:30:51.547838+0000","last_unstale":"2024-08-19T01:31:35.598931+0000","last_undegraded":"2024-08-19T01:31:35.598931+0000","last_fullsized":"2024-08-19T01:31:35.598931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:08:49.671964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595172+0000","last_change":"2024-08-19T01:30:49.576799+0000","last_active":"2024-08-19T01:31:35.595172+0000","last_peered":"2024-08-19T01:31:35.595172+0000","last_clean":"2024-08-19T01:31:35.595172+0000","last_became_active":"2024-08-19T01:30:49.576385+0000","last_became_peered":"2024-08-19T01:30:49.576385+0000","last_unstale":"2024-08-19T01:31:35.595172+0000","last_undegraded":"2024-08-19T01:31:35.595172+0000","last_fullsized":"2024-08-19T01:31:35.595172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:34:18.287518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603305+0000","last_change":"2024-08-19T01:30:58.593703+0000","last_active":"2024-08-19T01:31:35.603305+0000","last_peered":"2024-08-19T01:31:35.603305+0000","last_clean":"2024-08-19T01:31:35.603305+0000","last_became_active":"2024-08-19T01:30:58.593303+0000","last_became_peered":"2024-08-19T01:30:58.593303+0000","last_unstale":"2024-08-19T01:31:35.603305+0000","last_undegraded":"2024-08-19T01:31:35.603305+0000","last_fullsized":"2024-08-19T01:31:35.603305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:12:43.359262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601173+0000","last_change":"2024-08-19T01:31:35.601173+0000","last_active":"2024-08-19T01:31:35.601173+0000","last_peered":"2024-08-19T01:31:35.601173+0000","last_clean":"2024-08-19T01:31:35.601173+0000","last_became_active":"2024-08-19T01:31:35.600742+0000","last_became_peered":"2024-08-19T01:31:35.600742+0000","last_unstale":"2024-08-19T01:31:35.601173+0000","last_undegraded":"2024-08-19T01:31:35.601173+0000","last_fullsized":"2024-08-19T01:31:35.601173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:36:48.977124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604573+0000","last_change":"2024-08-19T01:30:49.540070+0000","last_active":"2024-08-19T01:31:35.604573+0000","last_peered":"2024-08-19T01:31:35.604573+0000","last_clean":"2024-08-19T01:31:35.604573+0000","last_became_active":"2024-08-19T01:30:49.539706+0000","last_became_peered":"2024-08-19T01:30:49.539706+0000","last_unstale":"2024-08-19T01:31:35.604573+0000","last_undegraded":"2024-08-19T01:31:35.604573+0000","last_fullsized":"2024-08-19T01:31:35.604573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:14:15.813670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745319+0000","last_change":"2024-08-19T01:30:58.590766+0000","last_active":"2024-08-19T01:31:35.745319+0000","last_peered":"2024-08-19T01:31:35.745319+0000","last_clean":"2024-08-19T01:31:35.745319+0000","last_became_active":"2024-08-19T01:30:58.590415+0000","last_became_peered":"2024-08-19T01:30:58.590415+0000","last_unstale":"2024-08-19T01:31:35.745319+0000","last_undegraded":"2024-08-19T01:31:35.745319+0000","last_fullsized":"2024-08-19T01:31:35.745319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:48:59.261086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.602761+0000","last_change":"2024-08-19T01:31:35.602332+0000","last_active":"2024-08-19T01:31:35.602761+0000","last_peered":"2024-08-19T01:31:35.602761+0000","last_clean":"2024-08-19T01:31:35.602761+0000","last_became_active":"2024-08-19T01:31:35.602007+0000","last_became_peered":"2024-08-19T01:31:35.602007+0000","last_unstale":"2024-08-19T01:31:35.602761+0000","last_undegraded":"2024-08-19T01:31:35.602761+0000","last_fullsized":"2024-08-19T01:31:35.602761+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:17:47.694567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:49.623474+0000","last_change":"2024-08-19T01:30:49.575995+0000","last_active":"2024-08-19T01:31:49.623474+0000","last_peered":"2024-08-19T01:31:49.623474+0000","last_clean":"2024-08-19T01:31:49.623474+0000","last_became_active":"2024-08-19T01:30:49.575249+0000","last_became_peered":"2024-08-19T01:30:49.575249+0000","last_unstale":"2024-08-19T01:31:49.623474+0000","last_undegraded":"2024-08-19T01:31:49.623474+0000","last_fullsized":"2024-08-19T01:31:49.623474+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:39:05.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745536+0000","last_change":"2024-08-19T01:30:58.598812+0000","last_active":"2024-08-19T01:31:35.745536+0000","last_peered":"2024-08-19T01:31:35.745536+0000","last_clean":"2024-08-19T01:31:35.745536+0000","last_became_active":"2024-08-19T01:30:58.598434+0000","last_became_peered":"2024-08-19T01:30:58.598434+0000","last_unstale":"2024-08-19T01:31:35.745536+0000","last_undegraded":"2024-08-19T01:31:35.745536+0000","last_fullsized":"2024-08-19T01:31:35.745536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:15:19.381503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744921+0000","last_change":"2024-08-19T01:30:51.549158+0000","last_active":"2024-08-19T01:31:35.744921+0000","last_peered":"2024-08-19T01:31:35.744921+0000","last_clean":"2024-08-19T01:31:35.744921+0000","last_became_active":"2024-08-19T01:30:51.548854+0000","last_became_peered":"2024-08-19T01:30:51.548854+0000","last_unstale":"2024-08-19T01:31:35.744921+0000","last_undegraded":"2024-08-19T01:31:35.744921+0000","last_fullsized":"2024-08-19T01:31:35.744921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:35.661758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604219+0000","last_change":"2024-08-19T01:30:49.538282+0000","last_active":"2024-08-19T01:31:35.604219+0000","last_peered":"2024-08-19T01:31:35.604219+0000","last_clean":"2024-08-19T01:31:35.604219+0000","last_became_active":"2024-08-19T01:30:49.537918+0000","last_became_peered":"2024-08-19T01:30:49.537918+0000","last_unstale":"2024-08-19T01:31:35.604219+0000","last_undegraded":"2024-08-19T01:31:35.604219+0000","last_fullsized":"2024-08-19T01:31:35.604219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:27.886958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601012+0000","last_change":"2024-08-19T01:30:58.591153+0000","last_active":"2024-08-19T01:31:35.601012+0000","last_peered":"2024-08-19T01:31:35.601012+0000","last_clean":"2024-08-19T01:31:35.601012+0000","last_became_active":"2024-08-19T01:30:58.590738+0000","last_became_peered":"2024-08-19T01:30:58.590738+0000","last_unstale":"2024-08-19T01:31:35.601012+0000","last_undegraded":"2024-08-19T01:31:35.601012+0000","last_fullsized":"2024-08-19T01:31:35.601012+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:29.073825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868743+0000","last_change":"2024-08-19T01:30:51.554878+0000","last_active":"2024-08-19T01:31:35.868743+0000","last_peered":"2024-08-19T01:31:35.868743+0000","last_clean":"2024-08-19T01:31:35.868743+0000","last_became_active":"2024-08-19T01:30:51.553743+0000","last_became_peered":"2024-08-19T01:30:51.553743+0000","last_unstale":"2024-08-19T01:31:35.868743+0000","last_undegraded":"2024-08-19T01:31:35.868743+0000","last_fullsized":"2024-08-19T01:31:35.868743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:08:41.932641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600867+0000","last_change":"2024-08-19T01:30:49.542043+0000","last_active":"2024-08-19T01:31:35.600867+0000","last_peered":"2024-08-19T01:31:35.600867+0000","last_clean":"2024-08-19T01:31:35.600867+0000","last_became_active":"2024-08-19T01:30:49.541551+0000","last_became_peered":"2024-08-19T01:30:49.541551+0000","last_unstale":"2024-08-19T01:31:35.600867+0000","last_undegraded":"2024-08-19T01:31:35.600867+0000","last_fullsized":"2024-08-19T01:31:35.600867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:46:14.781516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":364,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604258+0000","last_change":"2024-08-19T01:29:41.574391+0000","last_active":"2024-08-19T01:31:35.604258+0000","last_peered":"2024-08-19T01:31:35.604258+0000","last_clean":"2024-08-19T01:31:35.604258+0000","last_became_active":"2024-08-19T01:29:41.573885+0000","last_became_peered":"2024-08-19T01:29:41.573885+0000","last_unstale":"2024-08-19T01:31:35.604258+0000","last_undegraded":"2024-08-19T01:31:35.604258+0000","last_fullsized":"2024-08-19T01:31:35.604258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_clean_scrub_stamp":"2024-08-19T01:26:37.926225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:55:38.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:58.598415+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:58.598086+0000","last_became_peered":"2024-08-19T01:30:58.598086+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:21:37.338869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605022+0000","last_change":"2024-08-19T01:30:51.560407+0000","last_active":"2024-08-19T01:31:35.605022+0000","last_peered":"2024-08-19T01:31:35.605022+0000","last_clean":"2024-08-19T01:31:35.605022+0000","last_became_active":"2024-08-19T01:30:51.559881+0000","last_became_peered":"2024-08-19T01:30:51.559881+0000","last_unstale":"2024-08-19T01:31:35.605022+0000","last_undegraded":"2024-08-19T01:31:35.605022+0000","last_fullsized":"2024-08-19T01:31:35.605022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:24:37.928254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597862+0000","last_change":"2024-08-19T01:30:49.711171+0000","last_active":"2024-08-19T01:31:35.597862+0000","last_peered":"2024-08-19T01:31:35.597862+0000","last_clean":"2024-08-19T01:31:35.597862+0000","last_became_active":"2024-08-19T01:30:49.710707+0000","last_became_peered":"2024-08-19T01:30:49.710707+0000","last_unstale":"2024-08-19T01:31:35.597862+0000","last_undegraded":"2024-08-19T01:31:35.597862+0000","last_fullsized":"2024-08-19T01:31:35.597862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:58:20.924502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595955+0000","last_change":"2024-08-19T01:30:58.590829+0000","last_active":"2024-08-19T01:31:35.595955+0000","last_peered":"2024-08-19T01:31:35.595955+0000","last_clean":"2024-08-19T01:31:35.595955+0000","last_became_active":"2024-08-19T01:30:58.590460+0000","last_became_peered":"2024-08-19T01:30:58.590460+0000","last_unstale":"2024-08-19T01:31:35.595955+0000","last_undegraded":"2024-08-19T01:31:35.595955+0000","last_fullsized":"2024-08-19T01:31:35.595955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:04.118550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604645+0000","last_change":"2024-08-19T01:30:51.564990+0000","last_active":"2024-08-19T01:31:35.604645+0000","last_peered":"2024-08-19T01:31:35.604645+0000","last_clean":"2024-08-19T01:31:35.604645+0000","last_became_active":"2024-08-19T01:30:51.564653+0000","last_became_peered":"2024-08-19T01:30:51.564653+0000","last_unstale":"2024-08-19T01:31:35.604645+0000","last_undegraded":"2024-08-19T01:31:35.604645+0000","last_fullsized":"2024-08-19T01:31:35.604645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:15:41.154376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601128+0000","last_change":"2024-08-19T01:30:49.537637+0000","last_active":"2024-08-19T01:31:35.601128+0000","last_peered":"2024-08-19T01:31:35.601128+0000","last_clean":"2024-08-19T01:31:35.601128+0000","last_became_active":"2024-08-19T01:30:49.537277+0000","last_became_peered":"2024-08-19T01:30:49.537277+0000","last_unstale":"2024-08-19T01:31:35.601128+0000","last_undegraded":"2024-08-19T01:31:35.601128+0000","last_fullsized":"2024-08-19T01:31:35.601128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:13.547597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598003+0000","last_change":"2024-08-19T01:30:58.594675+0000","last_active":"2024-08-19T01:31:35.598003+0000","last_peered":"2024-08-19T01:31:35.598003+0000","last_clean":"2024-08-19T01:31:35.598003+0000","last_became_active":"2024-08-19T01:30:58.594323+0000","last_became_peered":"2024-08-19T01:30:58.594323+0000","last_unstale":"2024-08-19T01:31:35.598003+0000","last_undegraded":"2024-08-19T01:31:35.598003+0000","last_fullsized":"2024-08-19T01:31:35.598003+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:21:05.447106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605468+0000","last_change":"2024-08-19T01:30:51.565014+0000","last_active":"2024-08-19T01:31:35.605468+0000","last_peered":"2024-08-19T01:31:35.605468+0000","last_clean":"2024-08-19T01:31:35.605468+0000","last_became_active":"2024-08-19T01:30:51.564672+0000","last_became_peered":"2024-08-19T01:30:51.564672+0000","last_unstale":"2024-08-19T01:31:35.605468+0000","last_undegraded":"2024-08-19T01:31:35.605468+0000","last_fullsized":"2024-08-19T01:31:35.605468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:56:07.404042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745571+0000","last_change":"2024-08-19T01:30:49.538423+0000","last_active":"2024-08-19T01:31:35.745571+0000","last_peered":"2024-08-19T01:31:35.745571+0000","last_clean":"2024-08-19T01:31:35.745571+0000","last_became_active":"2024-08-19T01:30:49.538049+0000","last_became_peered":"2024-08-19T01:30:49.538049+0000","last_unstale":"2024-08-19T01:31:35.745571+0000","last_undegraded":"2024-08-19T01:31:35.745571+0000","last_fullsized":"2024-08-19T01:31:35.745571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:27:41.449742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599789+0000","last_change":"2024-08-19T01:30:58.590963+0000","last_active":"2024-08-19T01:31:35.599789+0000","last_peered":"2024-08-19T01:31:35.599789+0000","last_clean":"2024-08-19T01:31:35.599789+0000","last_became_active":"2024-08-19T01:30:58.590584+0000","last_became_peered":"2024-08-19T01:30:58.590584+0000","last_unstale":"2024-08-19T01:31:35.599789+0000","last_undegraded":"2024-08-19T01:31:35.599789+0000","last_fullsized":"2024-08-19T01:31:35.599789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:37.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599809+0000","last_change":"2024-08-19T01:30:51.561298+0000","last_active":"2024-08-19T01:31:35.599809+0000","last_peered":"2024-08-19T01:31:35.599809+0000","last_clean":"2024-08-19T01:31:35.599809+0000","last_became_active":"2024-08-19T01:30:51.560919+0000","last_became_peered":"2024-08-19T01:30:51.560919+0000","last_unstale":"2024-08-19T01:31:35.599809+0000","last_undegraded":"2024-08-19T01:31:35.599809+0000","last_fullsized":"2024-08-19T01:31:35.599809+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:04:39.808838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604897+0000","last_change":"2024-08-19T01:30:49.541968+0000","last_active":"2024-08-19T01:31:35.604897+0000","last_peered":"2024-08-19T01:31:35.604897+0000","last_clean":"2024-08-19T01:31:35.604897+0000","last_became_active":"2024-08-19T01:30:49.541512+0000","last_became_peered":"2024-08-19T01:30:49.541512+0000","last_unstale":"2024-08-19T01:31:35.604897+0000","last_undegraded":"2024-08-19T01:31:35.604897+0000","last_fullsized":"2024-08-19T01:31:35.604897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:34:28.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600022+0000","last_change":"2024-08-19T01:30:58.594776+0000","last_active":"2024-08-19T01:31:35.600022+0000","last_peered":"2024-08-19T01:31:35.600022+0000","last_clean":"2024-08-19T01:31:35.600022+0000","last_became_active":"2024-08-19T01:30:58.594414+0000","last_became_peered":"2024-08-19T01:30:58.594414+0000","last_unstale":"2024-08-19T01:31:35.600022+0000","last_undegraded":"2024-08-19T01:31:35.600022+0000","last_fullsized":"2024-08-19T01:31:35.600022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:32:20.121504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600072+0000","last_change":"2024-08-19T01:30:51.556696+0000","last_active":"2024-08-19T01:31:35.600072+0000","last_peered":"2024-08-19T01:31:35.600072+0000","last_clean":"2024-08-19T01:31:35.600072+0000","last_became_active":"2024-08-19T01:30:51.555873+0000","last_became_peered":"2024-08-19T01:30:51.555873+0000","last_unstale":"2024-08-19T01:31:35.600072+0000","last_undegraded":"2024-08-19T01:31:35.600072+0000","last_fullsized":"2024-08-19T01:31:35.600072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:50:27.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603398+0000","last_change":"2024-08-19T01:30:49.576925+0000","last_active":"2024-08-19T01:31:35.603398+0000","last_peered":"2024-08-19T01:31:35.603398+0000","last_clean":"2024-08-19T01:31:35.603398+0000","last_became_active":"2024-08-19T01:30:49.576509+0000","last_became_peered":"2024-08-19T01:30:49.576509+0000","last_unstale":"2024-08-19T01:31:35.603398+0000","last_undegraded":"2024-08-19T01:31:35.603398+0000","last_fullsized":"2024-08-19T01:31:35.603398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:43:01.887020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745511+0000","last_change":"2024-08-19T01:30:58.590940+0000","last_active":"2024-08-19T01:31:35.745511+0000","last_peered":"2024-08-19T01:31:35.745511+0000","last_clean":"2024-08-19T01:31:35.745511+0000","last_became_active":"2024-08-19T01:30:58.590565+0000","last_became_peered":"2024-08-19T01:30:58.590565+0000","last_unstale":"2024-08-19T01:31:35.745511+0000","last_undegraded":"2024-08-19T01:31:35.745511+0000","last_fullsized":"2024-08-19T01:31:35.745511+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:38:31.345290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603937+0000","last_change":"2024-08-19T01:30:51.559622+0000","last_active":"2024-08-19T01:31:35.603937+0000","last_peered":"2024-08-19T01:31:35.603937+0000","last_clean":"2024-08-19T01:31:35.603937+0000","last_became_active":"2024-08-19T01:30:51.559263+0000","last_became_peered":"2024-08-19T01:30:51.559263+0000","last_unstale":"2024-08-19T01:31:35.603937+0000","last_undegraded":"2024-08-19T01:31:35.603937+0000","last_fullsized":"2024-08-19T01:31:35.603937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:11:25.758915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:49.623985+0000","last_change":"2024-08-19T01:30:49.540102+0000","last_active":"2024-08-19T01:31:49.623985+0000","last_peered":"2024-08-19T01:31:49.623985+0000","last_clean":"2024-08-19T01:31:49.623985+0000","last_became_active":"2024-08-19T01:30:49.539740+0000","last_became_peered":"2024-08-19T01:30:49.539740+0000","last_unstale":"2024-08-19T01:31:49.623985+0000","last_undegraded":"2024-08-19T01:31:49.623985+0000","last_fullsized":"2024-08-19T01:31:49.623985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:47:21.684925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595191+0000","last_change":"2024-08-19T01:30:58.595748+0000","last_active":"2024-08-19T01:31:35.595191+0000","last_peered":"2024-08-19T01:31:35.595191+0000","last_clean":"2024-08-19T01:31:35.595191+0000","last_became_active":"2024-08-19T01:30:58.595396+0000","last_became_peered":"2024-08-19T01:30:58.595396+0000","last_unstale":"2024-08-19T01:31:35.595191+0000","last_undegraded":"2024-08-19T01:31:35.595191+0000","last_fullsized":"2024-08-19T01:31:35.595191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:58:29.113687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604652+0000","last_change":"2024-08-19T01:30:51.559159+0000","last_active":"2024-08-19T01:31:35.604652+0000","last_peered":"2024-08-19T01:31:35.604652+0000","last_clean":"2024-08-19T01:31:35.604652+0000","last_became_active":"2024-08-19T01:30:51.558226+0000","last_became_peered":"2024-08-19T01:30:51.558226+0000","last_unstale":"2024-08-19T01:31:35.604652+0000","last_undegraded":"2024-08-19T01:31:35.604652+0000","last_fullsized":"2024-08-19T01:31:35.604652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:12:54.447301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600514+0000","last_change":"2024-08-19T01:30:49.540806+0000","last_active":"2024-08-19T01:31:35.600514+0000","last_peered":"2024-08-19T01:31:35.600514+0000","last_clean":"2024-08-19T01:31:35.600514+0000","last_became_active":"2024-08-19T01:30:49.540439+0000","last_became_peered":"2024-08-19T01:30:49.540439+0000","last_unstale":"2024-08-19T01:31:35.600514+0000","last_undegraded":"2024-08-19T01:31:35.600514+0000","last_fullsized":"2024-08-19T01:31:35.600514+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:40.088842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599613+0000","last_change":"2024-08-19T01:30:58.592661+0000","last_active":"2024-08-19T01:31:35.599613+0000","last_peered":"2024-08-19T01:31:35.599613+0000","last_clean":"2024-08-19T01:31:35.599613+0000","last_became_active":"2024-08-19T01:30:58.592066+0000","last_became_peered":"2024-08-19T01:30:58.592066+0000","last_unstale":"2024-08-19T01:31:35.599613+0000","last_undegraded":"2024-08-19T01:31:35.599613+0000","last_fullsized":"2024-08-19T01:31:35.599613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:35.039682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745099+0000","last_change":"2024-08-19T01:30:51.558497+0000","last_active":"2024-08-19T01:31:35.745099+0000","last_peered":"2024-08-19T01:31:35.745099+0000","last_clean":"2024-08-19T01:31:35.745099+0000","last_became_active":"2024-08-19T01:30:51.558185+0000","last_became_peered":"2024-08-19T01:30:51.558185+0000","last_unstale":"2024-08-19T01:31:35.745099+0000","last_undegraded":"2024-08-19T01:31:35.745099+0000","last_fullsized":"2024-08-19T01:31:35.745099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:30:42.766269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597725+0000","last_change":"2024-08-19T01:30:49.539112+0000","last_active":"2024-08-19T01:31:35.597725+0000","last_peered":"2024-08-19T01:31:35.597725+0000","last_clean":"2024-08-19T01:31:35.597725+0000","last_became_active":"2024-08-19T01:30:49.538798+0000","last_became_peered":"2024-08-19T01:30:49.538798+0000","last_unstale":"2024-08-19T01:31:35.597725+0000","last_undegraded":"2024-08-19T01:31:35.597725+0000","last_fullsized":"2024-08-19T01:31:35.597725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:10:58.977041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600778+0000","last_change":"2024-08-19T01:30:58.590727+0000","last_active":"2024-08-19T01:31:35.600778+0000","last_peered":"2024-08-19T01:31:35.600778+0000","last_clean":"2024-08-19T01:31:35.600778+0000","last_became_active":"2024-08-19T01:30:58.590208+0000","last_became_peered":"2024-08-19T01:30:58.590208+0000","last_unstale":"2024-08-19T01:31:35.600778+0000","last_undegraded":"2024-08-19T01:31:35.600778+0000","last_fullsized":"2024-08-19T01:31:35.600778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:36.422915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604177+0000","last_change":"2024-08-19T01:30:51.565113+0000","last_active":"2024-08-19T01:31:35.604177+0000","last_peered":"2024-08-19T01:31:35.604177+0000","last_clean":"2024-08-19T01:31:35.604177+0000","last_became_active":"2024-08-19T01:30:51.564769+0000","last_became_peered":"2024-08-19T01:30:51.564769+0000","last_unstale":"2024-08-19T01:31:35.604177+0000","last_undegraded":"2024-08-19T01:31:35.604177+0000","last_fullsized":"2024-08-19T01:31:35.604177+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:13:28.373114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'29","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:47.624786+0000","last_change":"2024-08-19T01:31:35.610319+0000","last_active":"2024-08-19T01:31:47.624786+0000","last_peered":"2024-08-19T01:31:47.624786+0000","last_clean":"2024-08-19T01:31:47.624786+0000","last_became_active":"2024-08-19T01:31:35.602527+0000","last_became_peered":"2024-08-19T01:31:35.602527+0000","last_unstale":"2024-08-19T01:31:47.624786+0000","last_undegraded":"2024-08-19T01:31:47.624786+0000","last_fullsized":"2024-08-19T01:31:47.624786+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:50:05.511332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56667,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":67,"num_write":30,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3841,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594820+0000","last_change":"2024-08-19T01:30:58.594836+0000","last_active":"2024-08-19T01:31:35.594820+0000","last_peered":"2024-08-19T01:31:35.594820+0000","last_clean":"2024-08-19T01:31:35.594820+0000","last_became_active":"2024-08-19T01:30:58.594441+0000","last_became_peered":"2024-08-19T01:30:58.594441+0000","last_unstale":"2024-08-19T01:31:35.594820+0000","last_undegraded":"2024-08-19T01:31:35.594820+0000","last_fullsized":"2024-08-19T01:31:35.594820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:29:16.482181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596226+0000","last_change":"2024-08-19T01:30:51.562567+0000","last_active":"2024-08-19T01:31:35.596226+0000","last_peered":"2024-08-19T01:31:35.596226+0000","last_clean":"2024-08-19T01:31:35.596226+0000","last_became_active":"2024-08-19T01:30:51.562211+0000","last_became_peered":"2024-08-19T01:30:51.562211+0000","last_unstale":"2024-08-19T01:31:35.596226+0000","last_undegraded":"2024-08-19T01:31:35.596226+0000","last_fullsized":"2024-08-19T01:31:35.596226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:58.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601097+0000","last_change":"2024-08-19T01:30:49.575957+0000","last_active":"2024-08-19T01:31:35.601097+0000","last_peered":"2024-08-19T01:31:35.601097+0000","last_clean":"2024-08-19T01:31:35.601097+0000","last_became_active":"2024-08-19T01:30:49.575570+0000","last_became_peered":"2024-08-19T01:30:49.575570+0000","last_unstale":"2024-08-19T01:31:35.601097+0000","last_undegraded":"2024-08-19T01:31:35.601097+0000","last_fullsized":"2024-08-19T01:31:35.601097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:47.038113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599852+0000","last_change":"2024-08-19T01:30:58.589830+0000","last_active":"2024-08-19T01:31:35.599852+0000","last_peered":"2024-08-19T01:31:35.599852+0000","last_clean":"2024-08-19T01:31:35.599852+0000","last_became_active":"2024-08-19T01:30:58.589292+0000","last_became_peered":"2024-08-19T01:30:58.589292+0000","last_unstale":"2024-08-19T01:31:35.599852+0000","last_undegraded":"2024-08-19T01:31:35.599852+0000","last_fullsized":"2024-08-19T01:31:35.599852+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:14.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.606449+0000","last_change":"2024-08-19T01:31:35.606449+0000","last_active":"2024-08-19T01:31:35.606449+0000","last_peered":"2024-08-19T01:31:35.606449+0000","last_clean":"2024-08-19T01:31:35.606449+0000","last_became_active":"2024-08-19T01:31:35.606105+0000","last_became_peered":"2024-08-19T01:31:35.606105+0000","last_unstale":"2024-08-19T01:31:35.606449+0000","last_undegraded":"2024-08-19T01:31:35.606449+0000","last_fullsized":"2024-08-19T01:31:35.606449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:36:21.834763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605342+0000","last_change":"2024-08-19T01:30:49.542441+0000","last_active":"2024-08-19T01:31:35.605342+0000","last_peered":"2024-08-19T01:31:35.605342+0000","last_clean":"2024-08-19T01:31:35.605342+0000","last_became_active":"2024-08-19T01:30:49.542106+0000","last_became_peered":"2024-08-19T01:30:49.542106+0000","last_unstale":"2024-08-19T01:31:35.605342+0000","last_undegraded":"2024-08-19T01:31:35.605342+0000","last_fullsized":"2024-08-19T01:31:35.605342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:21:01.956282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594574+0000","last_change":"2024-08-19T01:30:58.588899+0000","last_active":"2024-08-19T01:31:35.594574+0000","last_peered":"2024-08-19T01:31:35.594574+0000","last_clean":"2024-08-19T01:31:35.594574+0000","last_became_active":"2024-08-19T01:30:58.588521+0000","last_became_peered":"2024-08-19T01:30:58.588521+0000","last_unstale":"2024-08-19T01:31:35.594574+0000","last_undegraded":"2024-08-19T01:31:35.594574+0000","last_fullsized":"2024-08-19T01:31:35.594574+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:43:59.485456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596747+0000","last_change":"2024-08-19T01:30:51.557248+0000","last_active":"2024-08-19T01:31:35.596747+0000","last_peered":"2024-08-19T01:31:35.596747+0000","last_clean":"2024-08-19T01:31:35.596747+0000","last_became_active":"2024-08-19T01:30:51.556448+0000","last_became_peered":"2024-08-19T01:30:51.556448+0000","last_unstale":"2024-08-19T01:31:35.596747+0000","last_undegraded":"2024-08-19T01:31:35.596747+0000","last_fullsized":"2024-08-19T01:31:35.596747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:19:19.587955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:46.623075+0000","last_change":"2024-08-19T01:30:49.540360+0000","last_active":"2024-08-19T01:31:46.623075+0000","last_peered":"2024-08-19T01:31:46.623075+0000","last_clean":"2024-08-19T01:31:46.623075+0000","last_became_active":"2024-08-19T01:30:49.539993+0000","last_became_peered":"2024-08-19T01:30:49.539993+0000","last_unstale":"2024-08-19T01:31:46.623075+0000","last_undegraded":"2024-08-19T01:31:46.623075+0000","last_fullsized":"2024-08-19T01:31:46.623075+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:05:28.308441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868467+0000","last_change":"2024-08-19T01:30:58.592387+0000","last_active":"2024-08-19T01:31:35.868467+0000","last_peered":"2024-08-19T01:31:35.868467+0000","last_clean":"2024-08-19T01:31:35.868467+0000","last_became_active":"2024-08-19T01:30:58.592031+0000","last_became_peered":"2024-08-19T01:30:58.592031+0000","last_unstale":"2024-08-19T01:31:35.868467+0000","last_undegraded":"2024-08-19T01:31:35.868467+0000","last_fullsized":"2024-08-19T01:31:35.868467+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:31:23.159012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598112+0000","last_change":"2024-08-19T01:30:58.595650+0000","last_active":"2024-08-19T01:31:35.598112+0000","last_peered":"2024-08-19T01:31:35.598112+0000","last_clean":"2024-08-19T01:31:35.598112+0000","last_became_active":"2024-08-19T01:30:58.595304+0000","last_became_peered":"2024-08-19T01:30:58.595304+0000","last_unstale":"2024-08-19T01:31:35.598112+0000","last_undegraded":"2024-08-19T01:31:35.598112+0000","last_fullsized":"2024-08-19T01:31:35.598112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:23:53.517355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594771+0000","last_change":"2024-08-19T01:30:51.548683+0000","last_active":"2024-08-19T01:31:35.594771+0000","last_peered":"2024-08-19T01:31:35.594771+0000","last_clean":"2024-08-19T01:31:35.594771+0000","last_became_active":"2024-08-19T01:30:51.548382+0000","last_became_peered":"2024-08-19T01:30:51.548382+0000","last_unstale":"2024-08-19T01:31:35.594771+0000","last_undegraded":"2024-08-19T01:31:35.594771+0000","last_fullsized":"2024-08-19T01:31:35.594771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:46:37.551017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601362+0000","last_change":"2024-08-19T01:30:49.540136+0000","last_active":"2024-08-19T01:31:35.601362+0000","last_peered":"2024-08-19T01:31:35.601362+0000","last_clean":"2024-08-19T01:31:35.601362+0000","last_became_active":"2024-08-19T01:30:49.539680+0000","last_became_peered":"2024-08-19T01:30:49.539680+0000","last_unstale":"2024-08-19T01:31:35.601362+0000","last_undegraded":"2024-08-19T01:31:35.601362+0000","last_fullsized":"2024-08-19T01:31:35.601362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:25.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599759+0000","last_change":"2024-08-19T01:30:58.597130+0000","last_active":"2024-08-19T01:31:35.599759+0000","last_peered":"2024-08-19T01:31:35.599759+0000","last_clean":"2024-08-19T01:31:35.599759+0000","last_became_active":"2024-08-19T01:30:58.596834+0000","last_became_peered":"2024-08-19T01:30:58.596834+0000","last_unstale":"2024-08-19T01:31:35.599759+0000","last_undegraded":"2024-08-19T01:31:35.599759+0000","last_fullsized":"2024-08-19T01:31:35.599759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:13:10.441421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868410+0000","last_change":"2024-08-19T01:30:51.555575+0000","last_active":"2024-08-19T01:31:35.868410+0000","last_peered":"2024-08-19T01:31:35.868410+0000","last_clean":"2024-08-19T01:31:35.868410+0000","last_became_active":"2024-08-19T01:30:51.555309+0000","last_became_peered":"2024-08-19T01:30:51.555309+0000","last_unstale":"2024-08-19T01:31:35.868410+0000","last_undegraded":"2024-08-19T01:31:35.868410+0000","last_fullsized":"2024-08-19T01:31:35.868410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:11:25.515286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744964+0000","last_change":"2024-08-19T01:30:49.536844+0000","last_active":"2024-08-19T01:31:35.744964+0000","last_peered":"2024-08-19T01:31:35.744964+0000","last_clean":"2024-08-19T01:31:35.744964+0000","last_became_active":"2024-08-19T01:30:49.536496+0000","last_became_peered":"2024-08-19T01:30:49.536496+0000","last_unstale":"2024-08-19T01:31:35.744964+0000","last_undegraded":"2024-08-19T01:31:35.744964+0000","last_fullsized":"2024-08-19T01:31:35.744964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:04.917188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598260+0000","last_change":"2024-08-19T01:30:49.539356+0000","last_active":"2024-08-19T01:31:35.598260+0000","last_peered":"2024-08-19T01:31:35.598260+0000","last_clean":"2024-08-19T01:31:35.598260+0000","last_became_active":"2024-08-19T01:30:49.538953+0000","last_became_peered":"2024-08-19T01:30:49.538953+0000","last_unstale":"2024-08-19T01:31:35.598260+0000","last_undegraded":"2024-08-19T01:31:35.598260+0000","last_fullsized":"2024-08-19T01:31:35.598260+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:13:17.845140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.870021+0000","last_change":"2024-08-19T01:31:35.870021+0000","last_active":"2024-08-19T01:31:35.870021+0000","last_peered":"2024-08-19T01:31:35.870021+0000","last_clean":"2024-08-19T01:31:35.870021+0000","last_became_active":"2024-08-19T01:31:35.869742+0000","last_became_peered":"2024-08-19T01:31:35.869742+0000","last_unstale":"2024-08-19T01:31:35.870021+0000","last_undegraded":"2024-08-19T01:31:35.870021+0000","last_fullsized":"2024-08-19T01:31:35.870021+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:51.848694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597559+0000","last_change":"2024-08-19T01:30:58.590385+0000","last_active":"2024-08-19T01:31:35.597559+0000","last_peered":"2024-08-19T01:31:35.597559+0000","last_clean":"2024-08-19T01:31:35.597559+0000","last_became_active":"2024-08-19T01:30:58.590004+0000","last_became_peered":"2024-08-19T01:30:58.590004+0000","last_unstale":"2024-08-19T01:31:35.597559+0000","last_undegraded":"2024-08-19T01:31:35.597559+0000","last_fullsized":"2024-08-19T01:31:35.597559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:13:01.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:46.623494+0000","last_change":"2024-08-19T01:31:35.614962+0000","last_active":"2024-08-19T01:31:46.623494+0000","last_peered":"2024-08-19T01:31:46.623494+0000","last_clean":"2024-08-19T01:31:46.623494+0000","last_became_active":"2024-08-19T01:31:35.605904+0000","last_became_peered":"2024-08-19T01:31:35.605904+0000","last_unstale":"2024-08-19T01:31:46.623494+0000","last_undegraded":"2024-08-19T01:31:46.623494+0000","last_fullsized":"2024-08-19T01:31:46.623494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:26:36.102805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603484+0000","last_change":"2024-08-19T01:30:51.550448+0000","last_active":"2024-08-19T01:31:35.603484+0000","last_peered":"2024-08-19T01:31:35.603484+0000","last_clean":"2024-08-19T01:31:35.603484+0000","last_became_active":"2024-08-19T01:30:51.550074+0000","last_became_peered":"2024-08-19T01:30:51.550074+0000","last_unstale":"2024-08-19T01:31:35.603484+0000","last_undegraded":"2024-08-19T01:31:35.603484+0000","last_fullsized":"2024-08-19T01:31:35.603484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:33:13.016183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597794+0000","last_change":"2024-08-19T01:30:58.592077+0000","last_active":"2024-08-19T01:31:35.597794+0000","last_peered":"2024-08-19T01:31:35.597794+0000","last_clean":"2024-08-19T01:31:35.597794+0000","last_became_active":"2024-08-19T01:30:58.591651+0000","last_became_peered":"2024-08-19T01:30:58.591651+0000","last_unstale":"2024-08-19T01:31:35.597794+0000","last_undegraded":"2024-08-19T01:31:35.597794+0000","last_fullsized":"2024-08-19T01:31:35.597794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:09:24.980954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599538+0000","last_change":"2024-08-19T01:30:51.552772+0000","last_active":"2024-08-19T01:31:35.599538+0000","last_peered":"2024-08-19T01:31:35.599538+0000","last_clean":"2024-08-19T01:31:35.599538+0000","last_became_active":"2024-08-19T01:30:51.550904+0000","last_became_peered":"2024-08-19T01:30:51.550904+0000","last_unstale":"2024-08-19T01:31:35.599538+0000","last_undegraded":"2024-08-19T01:31:35.599538+0000","last_fullsized":"2024-08-19T01:31:35.599538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:21:50.082587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598023+0000","last_change":"2024-08-19T01:30:51.559756+0000","last_active":"2024-08-19T01:31:35.598023+0000","last_peered":"2024-08-19T01:31:35.598023+0000","last_clean":"2024-08-19T01:31:35.598023+0000","last_became_active":"2024-08-19T01:30:51.559370+0000","last_became_peered":"2024-08-19T01:30:51.559370+0000","last_unstale":"2024-08-19T01:31:35.598023+0000","last_undegraded":"2024-08-19T01:31:35.598023+0000","last_fullsized":"2024-08-19T01:31:35.598023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:27:30.525346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597386+0000","last_change":"2024-08-19T01:30:51.556577+0000","last_active":"2024-08-19T01:31:35.597386+0000","last_peered":"2024-08-19T01:31:35.597386+0000","last_clean":"2024-08-19T01:31:35.597386+0000","last_became_active":"2024-08-19T01:30:51.555929+0000","last_became_peered":"2024-08-19T01:30:51.555929+0000","last_unstale":"2024-08-19T01:31:35.597386+0000","last_undegraded":"2024-08-19T01:31:35.597386+0000","last_fullsized":"2024-08-19T01:31:35.597386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:46:31.660752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598037+0000","last_change":"2024-08-19T01:30:51.558626+0000","last_active":"2024-08-19T01:31:35.598037+0000","last_peered":"2024-08-19T01:31:35.598037+0000","last_clean":"2024-08-19T01:31:35.598037+0000","last_became_active":"2024-08-19T01:30:51.558188+0000","last_became_peered":"2024-08-19T01:30:51.558188+0000","last_unstale":"2024-08-19T01:31:35.598037+0000","last_undegraded":"2024-08-19T01:31:35.598037+0000","last_fullsized":"2024-08-19T01:31:35.598037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:43:44.777643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745491+0000","last_change":"2024-08-19T01:30:51.552870+0000","last_active":"2024-08-19T01:31:35.745491+0000","last_peered":"2024-08-19T01:31:35.745491+0000","last_clean":"2024-08-19T01:31:35.745491+0000","last_became_active":"2024-08-19T01:30:51.552514+0000","last_became_peered":"2024-08-19T01:30:51.552514+0000","last_unstale":"2024-08-19T01:31:35.745491+0000","last_undegraded":"2024-08-19T01:31:35.745491+0000","last_fullsized":"2024-08-19T01:31:35.745491+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:10:58.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600636+0000","last_change":"2024-08-19T01:30:51.554216+0000","last_active":"2024-08-19T01:31:35.600636+0000","last_peered":"2024-08-19T01:31:35.600636+0000","last_clean":"2024-08-19T01:31:35.600636+0000","last_became_active":"2024-08-19T01:30:51.552535+0000","last_became_peered":"2024-08-19T01:30:51.552535+0000","last_unstale":"2024-08-19T01:31:35.600636+0000","last_undegraded":"2024-08-19T01:31:35.600636+0000","last_fullsized":"2024-08-19T01:31:35.600636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:26:52.971897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598164+0000","last_change":"2024-08-19T01:30:51.561703+0000","last_active":"2024-08-19T01:31:35.598164+0000","last_peered":"2024-08-19T01:31:35.598164+0000","last_clean":"2024-08-19T01:31:35.598164+0000","last_became_active":"2024-08-19T01:30:51.561385+0000","last_became_peered":"2024-08-19T01:30:51.561385+0000","last_unstale":"2024-08-19T01:31:35.598164+0000","last_undegraded":"2024-08-19T01:31:35.598164+0000","last_fullsized":"2024-08-19T01:31:35.598164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:07:23.187023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868940+0000","last_change":"2024-08-19T01:30:51.564047+0000","last_active":"2024-08-19T01:31:35.868940+0000","last_peered":"2024-08-19T01:31:35.868940+0000","last_clean":"2024-08-19T01:31:35.868940+0000","last_became_active":"2024-08-19T01:30:51.563743+0000","last_became_peered":"2024-08-19T01:30:51.563743+0000","last_unstale":"2024-08-19T01:31:35.868940+0000","last_undegraded":"2024-08-19T01:31:35.868940+0000","last_fullsized":"2024-08-19T01:31:35.868940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:50:37.639496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601317+0000","last_change":"2024-08-19T01:30:51.557058+0000","last_active":"2024-08-19T01:31:35.601317+0000","last_peered":"2024-08-19T01:31:35.601317+0000","last_clean":"2024-08-19T01:31:35.601317+0000","last_became_active":"2024-08-19T01:30:51.554611+0000","last_became_peered":"2024-08-19T01:30:51.554611+0000","last_unstale":"2024-08-19T01:31:35.601317+0000","last_undegraded":"2024-08-19T01:31:35.601317+0000","last_fullsized":"2024-08-19T01:31:35.601317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:56:25.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868614+0000","last_change":"2024-08-19T01:30:51.564119+0000","last_active":"2024-08-19T01:31:35.868614+0000","last_peered":"2024-08-19T01:31:35.868614+0000","last_clean":"2024-08-19T01:31:35.868614+0000","last_became_active":"2024-08-19T01:30:51.563840+0000","last_became_peered":"2024-08-19T01:30:51.563840+0000","last_unstale":"2024-08-19T01:31:35.868614+0000","last_undegraded":"2024-08-19T01:31:35.868614+0000","last_fullsized":"2024-08-19T01:31:35.868614+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:44:44.928760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596119+0000","last_change":"2024-08-19T01:30:51.561639+0000","last_active":"2024-08-19T01:31:35.596119+0000","last_peered":"2024-08-19T01:31:35.596119+0000","last_clean":"2024-08-19T01:31:35.596119+0000","last_became_active":"2024-08-19T01:30:51.561207+0000","last_became_peered":"2024-08-19T01:30:51.561207+0000","last_unstale":"2024-08-19T01:31:35.596119+0000","last_undegraded":"2024-08-19T01:31:35.596119+0000","last_fullsized":"2024-08-19T01:31:35.596119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:12:20.076628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745343+0000","last_change":"2024-08-19T01:30:51.556700+0000","last_active":"2024-08-19T01:31:35.745343+0000","last_peered":"2024-08-19T01:31:35.745343+0000","last_clean":"2024-08-19T01:31:35.745343+0000","last_became_active":"2024-08-19T01:30:51.556407+0000","last_became_peered":"2024-08-19T01:30:51.556407+0000","last_unstale":"2024-08-19T01:31:35.745343+0000","last_undegraded":"2024-08-19T01:31:35.745343+0000","last_fullsized":"2024-08-19T01:31:35.745343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:03:16.664253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599693+0000","last_change":"2024-08-19T01:30:51.553764+0000","last_active":"2024-08-19T01:31:35.599693+0000","last_peered":"2024-08-19T01:31:35.599693+0000","last_clean":"2024-08-19T01:31:35.599693+0000","last_became_active":"2024-08-19T01:30:51.553442+0000","last_became_peered":"2024-08-19T01:30:51.553442+0000","last_unstale":"2024-08-19T01:31:35.599693+0000","last_undegraded":"2024-08-19T01:31:35.599693+0000","last_fullsized":"2024-08-19T01:31:35.599693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:36:44.476979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598004+0000","last_change":"2024-08-19T01:30:51.565528+0000","last_active":"2024-08-19T01:31:35.598004+0000","last_peered":"2024-08-19T01:31:35.598004+0000","last_clean":"2024-08-19T01:31:35.598004+0000","last_became_active":"2024-08-19T01:30:51.564649+0000","last_became_peered":"2024-08-19T01:30:51.564649+0000","last_unstale":"2024-08-19T01:31:35.598004+0000","last_undegraded":"2024-08-19T01:31:35.598004+0000","last_fullsized":"2024-08-19T01:31:35.598004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:04:12.868496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595505+0000","last_change":"2024-08-19T01:30:51.553425+0000","last_active":"2024-08-19T01:31:35.595505+0000","last_peered":"2024-08-19T01:31:35.595505+0000","last_clean":"2024-08-19T01:31:35.595505+0000","last_became_active":"2024-08-19T01:30:51.553123+0000","last_became_peered":"2024-08-19T01:30:51.553123+0000","last_unstale":"2024-08-19T01:31:35.595505+0000","last_undegraded":"2024-08-19T01:31:35.595505+0000","last_fullsized":"2024-08-19T01:31:35.595505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:07.969228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600659+0000","last_change":"2024-08-19T01:30:51.556820+0000","last_active":"2024-08-19T01:31:35.600659+0000","last_peered":"2024-08-19T01:31:35.600659+0000","last_clean":"2024-08-19T01:31:35.600659+0000","last_became_active":"2024-08-19T01:30:51.556532+0000","last_became_peered":"2024-08-19T01:30:51.556532+0000","last_unstale":"2024-08-19T01:31:35.600659+0000","last_undegraded":"2024-08-19T01:31:35.600659+0000","last_fullsized":"2024-08-19T01:31:35.600659+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:40:37.053193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595799+0000","last_change":"2024-08-19T01:30:51.561610+0000","last_active":"2024-08-19T01:31:35.595799+0000","last_peered":"2024-08-19T01:31:35.595799+0000","last_clean":"2024-08-19T01:31:35.595799+0000","last_became_active":"2024-08-19T01:30:51.561270+0000","last_became_peered":"2024-08-19T01:30:51.561270+0000","last_unstale":"2024-08-19T01:31:35.595799+0000","last_undegraded":"2024-08-19T01:31:35.595799+0000","last_fullsized":"2024-08-19T01:31:35.595799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:15:15.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597839+0000","last_change":"2024-08-19T01:30:51.558128+0000","last_active":"2024-08-19T01:31:35.597839+0000","last_peered":"2024-08-19T01:31:35.597839+0000","last_clean":"2024-08-19T01:31:35.597839+0000","last_became_active":"2024-08-19T01:30:51.557785+0000","last_became_peered":"2024-08-19T01:30:51.557785+0000","last_unstale":"2024-08-19T01:31:35.597839+0000","last_undegraded":"2024-08-19T01:31:35.597839+0000","last_fullsized":"2024-08-19T01:31:35.597839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:14.849212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597900+0000","last_change":"2024-08-19T01:30:51.553917+0000","last_active":"2024-08-19T01:31:35.597900+0000","last_peered":"2024-08-19T01:31:35.597900+0000","last_clean":"2024-08-19T01:31:35.597900+0000","last_became_active":"2024-08-19T01:30:51.553596+0000","last_became_peered":"2024-08-19T01:30:51.553596+0000","last_unstale":"2024-08-19T01:31:35.597900+0000","last_undegraded":"2024-08-19T01:31:35.597900+0000","last_fullsized":"2024-08-19T01:31:35.597900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:03:36.001486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745336+0000","last_change":"2024-08-19T01:30:51.555201+0000","last_active":"2024-08-19T01:31:35.745336+0000","last_peered":"2024-08-19T01:31:35.745336+0000","last_clean":"2024-08-19T01:31:35.745336+0000","last_became_active":"2024-08-19T01:30:51.554160+0000","last_became_peered":"2024-08-19T01:30:51.554160+0000","last_unstale":"2024-08-19T01:31:35.745336+0000","last_undegraded":"2024-08-19T01:31:35.745336+0000","last_fullsized":"2024-08-19T01:31:35.745336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:58:20.586931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600619+0000","last_change":"2024-08-19T01:30:51.560517+0000","last_active":"2024-08-19T01:31:35.600619+0000","last_peered":"2024-08-19T01:31:35.600619+0000","last_clean":"2024-08-19T01:31:35.600619+0000","last_became_active":"2024-08-19T01:30:51.557454+0000","last_became_peered":"2024-08-19T01:30:51.557454+0000","last_unstale":"2024-08-19T01:31:35.600619+0000","last_undegraded":"2024-08-19T01:31:35.600619+0000","last_fullsized":"2024-08-19T01:31:35.600619+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:41:08.192165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:51.565961+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:51.565491+0000","last_became_peered":"2024-08-19T01:30:51.565491+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:53:06.263411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599327+0000","last_change":"2024-08-19T01:30:51.551297+0000","last_active":"2024-08-19T01:31:35.599327+0000","last_peered":"2024-08-19T01:31:35.599327+0000","last_clean":"2024-08-19T01:31:35.599327+0000","last_became_active":"2024-08-19T01:30:51.550929+0000","last_became_peered":"2024-08-19T01:30:51.550929+0000","last_unstale":"2024-08-19T01:31:35.599327+0000","last_undegraded":"2024-08-19T01:31:35.599327+0000","last_fullsized":"2024-08-19T01:31:35.599327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:07:03.625360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603232+0000","last_change":"2024-08-19T01:31:35.602883+0000","last_active":"2024-08-19T01:31:35.603232+0000","last_peered":"2024-08-19T01:31:35.603232+0000","last_clean":"2024-08-19T01:31:35.603232+0000","last_became_active":"2024-08-19T01:31:35.602571+0000","last_became_peered":"2024-08-19T01:31:35.602571+0000","last_unstale":"2024-08-19T01:31:35.603232+0000","last_undegraded":"2024-08-19T01:31:35.603232+0000","last_fullsized":"2024-08-19T01:31:35.603232+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:08:13.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868405+0000","last_change":"2024-08-19T01:30:51.552548+0000","last_active":"2024-08-19T01:31:35.868405+0000","last_peered":"2024-08-19T01:31:35.868405+0000","last_clean":"2024-08-19T01:31:35.868405+0000","last_became_active":"2024-08-19T01:30:51.552264+0000","last_became_peered":"2024-08-19T01:30:51.552264+0000","last_unstale":"2024-08-19T01:31:35.868405+0000","last_undegraded":"2024-08-19T01:31:35.868405+0000","last_fullsized":"2024-08-19T01:31:35.868405+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:17:53.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603728+0000","last_change":"2024-08-19T01:30:51.558686+0000","last_active":"2024-08-19T01:31:35.603728+0000","last_peered":"2024-08-19T01:31:35.603728+0000","last_clean":"2024-08-19T01:31:35.603728+0000","last_became_active":"2024-08-19T01:30:51.558190+0000","last_became_peered":"2024-08-19T01:30:51.558190+0000","last_unstale":"2024-08-19T01:31:35.603728+0000","last_undegraded":"2024-08-19T01:31:35.603728+0000","last_fullsized":"2024-08-19T01:31:35.603728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:52:33.056948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598221+0000","last_change":"2024-08-19T01:30:51.548868+0000","last_active":"2024-08-19T01:31:35.598221+0000","last_peered":"2024-08-19T01:31:35.598221+0000","last_clean":"2024-08-19T01:31:35.598221+0000","last_became_active":"2024-08-19T01:30:51.548347+0000","last_became_peered":"2024-08-19T01:30:51.548347+0000","last_unstale":"2024-08-19T01:31:35.598221+0000","last_undegraded":"2024-08-19T01:31:35.598221+0000","last_fullsized":"2024-08-19T01:31:35.598221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:39:22.049290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603369+0000","last_change":"2024-08-19T01:30:51.554054+0000","last_active":"2024-08-19T01:31:35.603369+0000","last_peered":"2024-08-19T01:31:35.603369+0000","last_clean":"2024-08-19T01:31:35.603369+0000","last_became_active":"2024-08-19T01:30:51.553731+0000","last_became_peered":"2024-08-19T01:30:51.553731+0000","last_unstale":"2024-08-19T01:31:35.603369+0000","last_undegraded":"2024-08-19T01:31:35.603369+0000","last_fullsized":"2024-08-19T01:31:35.603369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:25:17.939024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600261+0000","last_change":"2024-08-19T01:30:51.558662+0000","last_active":"2024-08-19T01:31:35.600261+0000","last_peered":"2024-08-19T01:31:35.600261+0000","last_clean":"2024-08-19T01:31:35.600261+0000","last_became_active":"2024-08-19T01:30:51.558131+0000","last_became_peered":"2024-08-19T01:30:51.558131+0000","last_unstale":"2024-08-19T01:31:35.600261+0000","last_undegraded":"2024-08-19T01:31:35.600261+0000","last_fullsized":"2024-08-19T01:31:35.600261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:31:25.054005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598117+0000","last_change":"2024-08-19T01:30:51.565170+0000","last_active":"2024-08-19T01:31:35.598117+0000","last_peered":"2024-08-19T01:31:35.598117+0000","last_clean":"2024-08-19T01:31:35.598117+0000","last_became_active":"2024-08-19T01:30:51.564765+0000","last_became_peered":"2024-08-19T01:30:51.564765+0000","last_unstale":"2024-08-19T01:31:35.598117+0000","last_undegraded":"2024-08-19T01:31:35.598117+0000","last_fullsized":"2024-08-19T01:31:35.598117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:00:48.668865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598106+0000","last_change":"2024-08-19T01:30:51.555300+0000","last_active":"2024-08-19T01:31:35.598106+0000","last_peered":"2024-08-19T01:31:35.598106+0000","last_clean":"2024-08-19T01:31:35.598106+0000","last_became_active":"2024-08-19T01:30:51.554977+0000","last_became_peered":"2024-08-19T01:30:51.554977+0000","last_unstale":"2024-08-19T01:31:35.598106+0000","last_undegraded":"2024-08-19T01:31:35.598106+0000","last_fullsized":"2024-08-19T01:31:35.598106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:18:58.961783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600410+0000","last_change":"2024-08-19T01:30:51.557906+0000","last_active":"2024-08-19T01:31:35.600410+0000","last_peered":"2024-08-19T01:31:35.600410+0000","last_clean":"2024-08-19T01:31:35.600410+0000","last_became_active":"2024-08-19T01:30:51.557598+0000","last_became_peered":"2024-08-19T01:30:51.557598+0000","last_unstale":"2024-08-19T01:31:35.600410+0000","last_undegraded":"2024-08-19T01:31:35.600410+0000","last_fullsized":"2024-08-19T01:31:35.600410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:33:12.785910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61057,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":110,"num_read_kb":115,"num_write":165,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4463,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282761,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481815,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":513466,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1157791,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906978,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":517798,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168613,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":513622,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430250,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1109920,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593776,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":508878,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822708,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1105118,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986233,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":570585,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215163,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":509072,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":509516,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":566467,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":510266,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":57679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T01:31:52.521 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-19T01:31:52.521 INFO:tasks.cephadm:Setting up client nodes... 2024-08-19T01:31:52.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-19T01:31:52.777 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/795845991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c18100800 msgr2=0x7f6c18100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/795845991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c18100800 0x7f6c18100c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009900 tx=0x7f6c1002a410 comp rx=0 tx=0).stop 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/795845991 shutdown_connections 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/795845991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c181023c0 0x7f6c18108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/795845991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c18101a00 0x7f6c18101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/795845991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c18100800 0x7f6c18100c00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/795845991 >> 172.21.15.64:0/795845991 conn(0x7f6c180fbfb0 msgr2=0x7f6c180fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/795845991 shutdown_connections 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.277+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/795845991 wait complete. 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 Processor -- start 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 -- start start 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c18100800 0x7f6c18196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c18101a00 0x7f6c181970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c181023c0 0x7f6c1819e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1810c500 con 0x7f6c181023c0 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1810c380 con 0x7f6c18100800 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1ca6f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1810c680 con 0x7f6c18101a00 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c181023c0 0x7f6c1819e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c181023c0 0x7f6c1819e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38426/0 (socket says 172.21.15.64:38426) 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 -- 172.21.15.64:0/387329930 learned_addr learned my addr 172.21.15.64:0/387329930 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1659c640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c18100800 0x7f6c18196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c15d9b640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c18101a00 0x7f6c181970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 -- 172.21.15.64:0/387329930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c18101a00 msgr2=0x7f6c181970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c18101a00 0x7f6c181970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 -- 172.21.15.64:0/387329930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c18100800 msgr2=0x7f6c18196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c18100800 0x7f6c18196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 -- 172.21.15.64:0/387329930 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1819e7e0 con 0x7f6c181023c0 2024-08-19T01:31:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c1659c640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c18100800 0x7f6c18196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.278+0000 7f6c16d9d640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c181023c0 0x7f6c1819e170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800b970 tx=0x7f6c0800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.279+0000 7f6c0f7fe640 1 -- 172.21.15.64:0/387329930 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08019070 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.279+0000 7f6c0f7fe640 1 -- 172.21.15.64:0/387329930 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c0800cb40 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.279+0000 7f6c0f7fe640 1 -- 172.21.15.64:0/387329930 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08007650 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.279+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1819ead0 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.279+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1819f010 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.280+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c18107c50 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.281+0000 7f6c0f7fe640 1 -- 172.21.15.64:0/387329930 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6c0801c070 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.282+0000 7f6c0f7fe640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6c040a2cc0 0x7f6c040a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.282+0000 7f6c1659c640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6c040a2cc0 0x7f6c040a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.282+0000 7f6c0f7fe640 1 -- 172.21.15.64:0/387329930 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6c080ce9d0 con 0x7f6c181023c0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.282+0000 7f6c1659c640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6c040a2cc0 0x7f6c040a5180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c10000c00 tx=0x7f6c10031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.286+0000 7f6c0f7fe640 1 -- 172.21.15.64:0/387329930 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6c0809b580 con 0x7f6c181023c0 2024-08-19T01:31:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.528+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f6c1802d830 con 0x7f6c181023c0 2024-08-19T01:31:53.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.530+0000 7f6c0f7fe640 1 -- 172.21.15.64:0/387329930 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) ==== 170+0+59 (secure 0 0 0) 0x7f6c0809ae80 con 0x7f6c181023c0 2024-08-19T01:31:53.532 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-08-19T01:31:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCJoMJmPVePHxAA6PE9T/onjBccoTb7U9AD2A== 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6c040a2cc0 msgr2=0x7f6c040a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6c040a2cc0 0x7f6c040a5180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c10000c00 tx=0x7f6c10031400 comp rx=0 tx=0).stop 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c181023c0 msgr2=0x7f6c1819e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c181023c0 0x7f6c1819e170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800b970 tx=0x7f6c0800be30 comp rx=0 tx=0).stop 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 shutdown_connections 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6c040a2cc0 0x7f6c040a5180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c181023c0 0x7f6c1819e170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c18101a00 0x7f6c181970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 --2- 172.21.15.64:0/387329930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c18100800 0x7f6c18196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:53.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 >> 172.21.15.64:0/387329930 conn(0x7f6c180fbfb0 msgr2=0x7f6c18104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:53.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 shutdown_connections 2024-08-19T01:31:53.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:53.532+0000 7f6c1ca6f640 1 -- 172.21.15.64:0/387329930 wait complete. 2024-08-19T01:31:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:53 smithi123 ceph-mon[21829]: pgmap v300: 129 pgs: 129 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.8 KiB/s wr, 8 op/s 2024-08-19T01:31:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:53 smithi123 ceph-mon[21829]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:31:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:53 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/387329930' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T01:31:53.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:53 smithi064 ceph-mon[18515]: pgmap v300: 129 pgs: 129 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.8 KiB/s wr, 8 op/s 2024-08-19T01:31:53.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:53 smithi064 ceph-mon[18515]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:31:53.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:53 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/387329930' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T01:31:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:53 smithi161 ceph-mon[21757]: pgmap v300: 129 pgs: 129 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.8 KiB/s wr, 8 op/s 2024-08-19T01:31:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:53 smithi161 ceph-mon[21757]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:31:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:53 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/387329930' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T01:31:54.291 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:31:54.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-08-19T01:31:54.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-19T01:31:54.330 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-19T01:31:54.330 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-19T01:31:54.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-19T01:31:54.630 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:54.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:54 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/387329930' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T01:31:54.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:54 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/387329930' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T01:31:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:54 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/387329930' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T01:31:55.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.101+0000 7f8281f1a640 1 -- 172.21.15.64:0/1399101029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c100790 msgr2=0x7f827c100bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.101+0000 7f8281f1a640 1 --2- 172.21.15.64:0/1399101029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c100790 0x7f827c100bf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8270009930 tx=0x7f827002a420 comp rx=0 tx=0).stop 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 -- 172.21.15.64:0/1399101029 shutdown_connections 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 --2- 172.21.15.64:0/1399101029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f827c101130 0x7f827c10d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 --2- 172.21.15.64:0/1399101029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c100790 0x7f827c100bf0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 --2- 172.21.15.64:0/1399101029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f827c106790 0x7f827c106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 -- 172.21.15.64:0/1399101029 >> 172.21.15.64:0/1399101029 conn(0x7f827c0fc4b0 msgr2=0x7f827c0fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 -- 172.21.15.64:0/1399101029 shutdown_connections 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 -- 172.21.15.64:0/1399101029 wait complete. 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 Processor -- start 2024-08-19T01:31:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.102+0000 7f8281f1a640 1 -- start start 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8281f1a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f827c100790 0x7f827c10e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8281f1a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f827c101130 0x7f827c10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8281f1a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c106790 0x7f827c10eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8281f1a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f827c111cb0 con 0x7f827c106790 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8281f1a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f827c111b30 con 0x7f827c101130 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8281f1a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f827c111e30 con 0x7f827c100790 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c106790 0x7f827c10eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c106790 0x7f827c10eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38434/0 (socket says 172.21.15.64:38434) 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 -- 172.21.15.64:0/3043020988 learned_addr learned my addr 172.21.15.64:0/3043020988 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827b7fe640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f827c100790 0x7f827c10e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827affd640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f827c101130 0x7f827c10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:55.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 -- 172.21.15.64:0/3043020988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f827c100790 msgr2=0x7f827c10e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f827c100790 0x7f827c10e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 -- 172.21.15.64:0/3043020988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f827c101130 msgr2=0x7f827c10e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f827c101130 0x7f827c10e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 -- 172.21.15.64:0/3043020988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827c1a66d0 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f827bfff640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c106790 0x7f827c10eb70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f827400b970 tx=0x7f827400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8278ff9640 1 -- 172.21.15.64:0/3043020988 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8274019070 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8278ff9640 1 -- 172.21.15.64:0/3043020988 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f827400cb40 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827c1a69c0 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.103+0000 7f8278ff9640 1 -- 172.21.15.64:0/3043020988 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8274007650 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.104+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f827c1a6ed0 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.104+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f827c100bf0 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.106+0000 7f8278ff9640 1 -- 172.21.15.64:0/3043020988 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f827401c070 con 0x7f827c106790 2024-08-19T01:31:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.106+0000 7f8278ff9640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f82640a2c10 0x7f82640a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:55.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.106+0000 7f827b7fe640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f82640a2c10 0x7f82640a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:55.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.106+0000 7f8278ff9640 1 -- 172.21.15.64:0/3043020988 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f82740ce980 con 0x7f827c106790 2024-08-19T01:31:55.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.107+0000 7f827b7fe640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f82640a2c10 0x7f82640a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f826c0058c0 tx=0x7f826c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:55.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.110+0000 7f8278ff9640 1 -- 172.21.15.64:0/3043020988 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f827409aee0 con 0x7f827c106790 2024-08-19T01:31:55.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.336+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f827c108d20 con 0x7f827c106790 2024-08-19T01:31:55.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.341+0000 7f8278ff9640 1 -- 172.21.15.64:0/3043020988 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263201 (secure 0 0 0) 0x7f827409f1e0 con 0x7f827c106790 2024-08-19T01:31:55.345 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:55.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f82640a2c10 msgr2=0x7f82640a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:55.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f82640a2c10 0x7f82640a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f826c0058c0 tx=0x7f826c00a2d0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c106790 msgr2=0x7f827c10eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:55.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c106790 0x7f827c10eb70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f827400b970 tx=0x7f827400be30 comp rx=0 tx=0).stop 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 shutdown_connections 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f82640a2c10 0x7f82640a50d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f827c106790 0x7f827c10eb70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f827c101130 0x7f827c10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 --2- 172.21.15.64:0/3043020988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f827c100790 0x7f827c10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 >> 172.21.15.64:0/3043020988 conn(0x7f827c0fc4b0 msgr2=0x7f827c0fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 shutdown_connections 2024-08-19T01:31:55.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:55.344+0000 7f8281f1a640 1 -- 172.21.15.64:0/3043020988 wait complete. 2024-08-19T01:31:55.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:55 smithi064 ceph-mon[18515]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.0 KiB/s wr, 9 op/s 2024-08-19T01:31:55.668 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:55 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3043020988' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-19T01:31:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:55 smithi123 ceph-mon[21829]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.0 KiB/s wr, 9 op/s 2024-08-19T01:31:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:55 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3043020988' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-19T01:31:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:55 smithi161 ceph-mon[21757]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.0 KiB/s wr, 9 op/s 2024-08-19T01:31:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:55 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3043020988' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-19T01:31:56.142 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2663741075},{"type":"v1","addr":"172.21.15.64:6801","nonce":2663741075}]},"active_addr":"172.21.15.64:6801/2663741075","active_change":"2024-08-19T01:24:33.597798+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.64:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1810737759}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1532949280}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1603406100}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1697666313}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2567890326}]}]} 2024-08-19T01:31:56.148 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-19T01:31:56.148 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-19T01:31:56.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:56.407 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 -- 172.21.15.64:0/2075923636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904100b50 msgr2=0x7f2904100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 --2- 172.21.15.64:0/2075923636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904100b50 0x7f2904100f30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009900 tx=0x7f28f402a480 comp rx=0 tx=0).stop 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 -- 172.21.15.64:0/2075923636 shutdown_connections 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 --2- 172.21.15.64:0/2075923636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2904102c20 0x7f290410d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 --2- 172.21.15.64:0/2075923636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29041022a0 0x7f29041026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 --2- 172.21.15.64:0/2075923636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904100b50 0x7f2904100f30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 -- 172.21.15.64:0/2075923636 >> 172.21.15.64:0/2075923636 conn(0x7f29040fc500 msgr2=0x7f29040fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.936+0000 7f2908ddd640 1 -- 172.21.15.64:0/2075923636 shutdown_connections 2024-08-19T01:31:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 -- 172.21.15.64:0/2075923636 wait complete. 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 Processor -- start 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 -- start start 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2904100b50 0x7f2904197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29041022a0 0x7f2904197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904102c20 0x7f290419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2904110cb0 con 0x7f2904102c20 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2904110b30 con 0x7f2904100b50 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.937+0000 7f2908ddd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2904110e30 con 0x7f29041022a0 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904102c20 0x7f290419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904102c20 0x7f290419ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38448/0 (socket says 172.21.15.64:38448) 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 -- 172.21.15.64:0/544581180 learned_addr learned my addr 172.21.15.64:0/544581180 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f290259c640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2904100b50 0x7f2904197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2901d9b640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29041022a0 0x7f2904197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 -- 172.21.15.64:0/544581180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29041022a0 msgr2=0x7f2904197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29041022a0 0x7f2904197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 -- 172.21.15.64:0/544581180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2904100b50 msgr2=0x7f2904197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2904100b50 0x7f2904197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 -- 172.21.15.64:0/544581180 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290419c0f0 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2901d9b640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29041022a0 0x7f2904197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f290259c640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2904100b50 0x7f2904197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2902d9d640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904102c20 0x7f290419ba10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00cce0 tx=0x7f28fc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f28f37fe640 1 -- 172.21.15.64:0/544581180 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc00f450 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f28f37fe640 1 -- 172.21.15.64:0/544581180 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28fc00fad0 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290419c3e0 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f28f37fe640 1 -- 172.21.15.64:0/544581180 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc018c10 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.938+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29041a3cc0 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.939+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2904100f30 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.940+0000 7f28f37fe640 1 -- 172.21.15.64:0/544581180 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f28fc00f5f0 con 0x7f2904102c20 2024-08-19T01:31:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.941+0000 7f28f37fe640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28ec0a2c10 0x7f28ec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.941+0000 7f290259c640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28ec0a2c10 0x7f28ec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.941+0000 7f28f37fe640 1 -- 172.21.15.64:0/544581180 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f28fc0cfc20 con 0x7f2904102c20 2024-08-19T01:31:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.942+0000 7f290259c640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28ec0a2c10 0x7f28ec0a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005ab0 tx=0x7f28f4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:56.945+0000 7f28f37fe640 1 -- 172.21.15.64:0/544581180 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f28fc014030 con 0x7f2904102c20 2024-08-19T01:31:57.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.121+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2904108830 con 0x7f2904102c20 2024-08-19T01:31:57.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.123+0000 7f28f37fe640 1 -- 172.21.15.64:0/544581180 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24643 (secure 0 0 0) 0x7f28fc09c6c0 con 0x7f2904102c20 2024-08-19T01:31:57.125 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:57.125 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":91,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:35.585064+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28ec0a2c10 msgr2=0x7f28ec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28ec0a2c10 0x7f28ec0a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005ab0 tx=0x7f28f4005a20 comp rx=0 tx=0).stop 2024-08-19T01:31:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904102c20 msgr2=0x7f290419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904102c20 0x7f290419ba10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00cce0 tx=0x7f28fc00a620 comp rx=0 tx=0).stop 2024-08-19T01:31:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 shutdown_connections 2024-08-19T01:31:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f28ec0a2c10 0x7f28ec0a50d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2904102c20 0x7f290419ba10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29041022a0 0x7f2904197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 --2- 172.21.15.64:0/544581180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2904100b50 0x7f2904197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 >> 172.21.15.64:0/544581180 conn(0x7f29040fc500 msgr2=0x7f29040fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 shutdown_connections 2024-08-19T01:31:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:57.125+0000 7f2908ddd640 1 -- 172.21.15.64:0/544581180 wait complete. 2024-08-19T01:31:57.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:57 smithi064 ceph-mon[18515]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-08-19T01:31:57.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:57 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/544581180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:57.867 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-19T01:31:57.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:31:57.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:57 smithi123 ceph-mon[21829]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-08-19T01:31:57.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:57 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/544581180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:57 smithi161 ceph-mon[21757]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-08-19T01:31:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:57 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/544581180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:58.122 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.603+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/852491958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec102c20 msgr2=0x7fe5ec10d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.603+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/852491958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec102c20 0x7fe5ec10d770 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009900 tx=0x7fe5dc02a480 comp rx=0 tx=0).stop 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/852491958 shutdown_connections 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/852491958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec102c20 0x7fe5ec10d770 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/852491958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec1022a0 0x7fe5ec1026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/852491958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec100b50 0x7fe5ec100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/852491958 >> 172.21.15.64:0/852491958 conn(0x7fe5ec0fc500 msgr2=0x7fe5ec0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:58.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/852491958 shutdown_connections 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/852491958 wait complete. 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 Processor -- start 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 -- start start 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec100b50 0x7fe5ec1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec1022a0 0x7fe5ec198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.604+0000 7fe5f0a5c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec102c20 0x7fe5ec19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5f0a5c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec111dd0 con 0x7fe5ec100b50 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5f0a5c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec111c50 con 0x7fe5ec102c20 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5f0a5c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec111f50 con 0x7fe5ec1022a0 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5ea59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec100b50 0x7fe5ec1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5ea59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec100b50 0x7fe5ec1981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38462/0 (socket says 172.21.15.64:38462) 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5ea59c640 1 -- 172.21.15.64:0/128060467 learned_addr learned my addr 172.21.15.64:0/128060467 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:31:58.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e9d9b640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec1022a0 0x7fe5ec198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5ead9d640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec102c20 0x7fe5ec19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e9d9b640 1 -- 172.21.15.64:0/128060467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec102c20 msgr2=0x7fe5ec19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e9d9b640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec102c20 0x7fe5ec19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e9d9b640 1 -- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec100b50 msgr2=0x7fe5ec1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e9d9b640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec100b50 0x7fe5ec1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e9d9b640 1 -- 172.21.15.64:0/128060467 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec19d180 con 0x7fe5ec1022a0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5ea59c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec100b50 0x7fe5ec1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5ead9d640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec102c20 0x7fe5ec19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e9d9b640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec1022a0 0x7fe5ec198710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e9f0 tx=0x7fe5d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e37fe640 1 -- 172.21.15.64:0/128060467 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d400ccc0 con 0x7fe5ec1022a0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e37fe640 1 -- 172.21.15.64:0/128060467 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5d4005020 con 0x7fe5ec1022a0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec19d470 con 0x7fe5ec1022a0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5e37fe640 1 -- 172.21.15.64:0/128060467 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d40049e0 con 0x7fe5ec1022a0 2024-08-19T01:31:58.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.605+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec1a4d50 con 0x7fe5ec1022a0 2024-08-19T01:31:58.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.606+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ec100f30 con 0x7fe5ec1022a0 2024-08-19T01:31:58.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.608+0000 7fe5e37fe640 1 -- 172.21.15.64:0/128060467 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe5d4005560 con 0x7fe5ec1022a0 2024-08-19T01:31:58.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.608+0000 7fe5e37fe640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5d80a2c10 0x7fe5d80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:31:58.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.608+0000 7fe5ea59c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5d80a2c10 0x7fe5d80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:31:58.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.608+0000 7fe5e37fe640 1 -- 172.21.15.64:0/128060467 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe5d40cf530 con 0x7fe5ec1022a0 2024-08-19T01:31:58.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.609+0000 7fe5ea59c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5d80a2c10 0x7fe5d80a50d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400f710 tx=0x7fe5e4007670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:31:58.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.612+0000 7fe5e37fe640 1 -- 172.21.15.64:0/128060467 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe5d40051d0 con 0x7fe5ec1022a0 2024-08-19T01:31:58.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.787+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5ec199530 con 0x7fe5ec1022a0 2024-08-19T01:31:58.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.789+0000 7fe5e37fe640 1 -- 172.21.15.64:0/128060467 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24643 (secure 0 0 0) 0x7fe5ec199530 con 0x7fe5ec1022a0 2024-08-19T01:31:58.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:31:58.791 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":91,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:31:35.585064+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5d80a2c10 msgr2=0x7fe5d80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5d80a2c10 0x7fe5d80a50d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400f710 tx=0x7fe5e4007670 comp rx=0 tx=0).stop 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec1022a0 msgr2=0x7fe5ec198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec1022a0 0x7fe5ec198710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e9f0 tx=0x7fe5d400eeb0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 shutdown_connections 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5d80a2c10 0x7fe5d80a50d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec102c20 0x7fe5ec19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec1022a0 0x7fe5ec198710 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 --2- 172.21.15.64:0/128060467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5ec100b50 0x7fe5ec1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 >> 172.21.15.64:0/128060467 conn(0x7fe5ec0fc500 msgr2=0x7fe5ec106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:31:58.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 shutdown_connections 2024-08-19T01:31:58.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:31:58.791+0000 7fe5f0a5c640 1 -- 172.21.15.64:0/128060467 wait complete. 2024-08-19T01:31:59.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-19T01:31:59.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-19T01:31:59.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-19T01:31:59.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-19T01:31:59.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-19T01:31:59.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-19T01:31:59.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-19T01:31:59.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-19T01:31:59.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-08-19T01:31:59.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-08-19T01:31:59.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-08-19T01:31:59.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-08-19T01:31:59.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:59 smithi064 ceph-mon[18515]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-08-19T01:31:59.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:31:59 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/128060467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:59.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:59 smithi123 ceph-mon[21829]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-08-19T01:31:59.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:31:59 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/128060467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:31:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:59 smithi161 ceph-mon[21757]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-08-19T01:31:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:31:59 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/128060467' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:32:00.285 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.286 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.286 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.286 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.292 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.294 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.294 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.299 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.304 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:00.309 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:01.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:01 smithi064 ceph-mon[18515]: pgmap v304: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-08-19T01:32:01.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:01 smithi123 ceph-mon[21829]: pgmap v304: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-08-19T01:32:01.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:01 smithi161 ceph-mon[21757]: pgmap v304: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-08-19T01:32:02.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 -- 172.21.15.64:0/2151962904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 msgr2=0x7fdc1c1026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 --2- 172.21.15.64:0/2151962904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 0x7fdc1c1026e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009900 tx=0x7fdc0402a480 comp rx=0 tx=0).stop 2024-08-19T01:32:02.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 -- 172.21.15.64:0/2151962904 shutdown_connections 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 --2- 172.21.15.64:0/2151962904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc1c102c20 0x7fdc1c10d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 --2- 172.21.15.64:0/2151962904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 0x7fdc1c1026e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 --2- 172.21.15.64:0/2151962904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c100b50 0x7fdc1c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 -- 172.21.15.64:0/2151962904 >> 172.21.15.64:0/2151962904 conn(0x7fdc1c0fc500 msgr2=0x7fdc1c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 -- 172.21.15.64:0/2151962904 shutdown_connections 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.506+0000 7fdc20a35640 1 -- 172.21.15.64:0/2151962904 wait complete. 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 Processor -- start 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 -- start start 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c100b50 0x7fdc1c197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 0x7fdc1c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc1c102c20 0x7fdc1c19ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c110cb0 con 0x7fdc1c1022a0 2024-08-19T01:32:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c110b30 con 0x7fdc1c100b50 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc20a35640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c110e30 con 0x7fdc1c102c20 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 0x7fdc1c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 0x7fdc1c197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42360/0 (socket says 172.21.15.64:42360) 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 -- 172.21.15.64:0/2398783933 learned_addr learned my addr 172.21.15.64:0/2398783933 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc1a59c640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c100b50 0x7fdc1c197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 -- 172.21.15.64:0/2398783933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc1c102c20 msgr2=0x7fdc1c19ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc1c102c20 0x7fdc1c19ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 -- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c100b50 msgr2=0x7fdc1c197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c100b50 0x7fdc1c197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.507+0000 7fdc19d9b640 1 -- 172.21.15.64:0/2398783933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c19c0e0 con 0x7fdc1c1022a0 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.508+0000 7fdc19d9b640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 0x7fdc1c197670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdc04006010 tx=0x7fdc04003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.508+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04035070 con 0x7fdc1c1022a0 2024-08-19T01:32:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.508+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c19c370 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.508+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc04031a20 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.508+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04038ae0 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.508+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c069ca0 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.509+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc1c06a250 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.510+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdc04031bd0 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc137fe640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdc080a2c10 0x7fdc080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc1a59c640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdc080a2c10 0x7fdc080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdc040f1900 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc137fe640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] conn(0x7fdc080adde0 0x7fdc080b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 --> [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc04049e50 con 0x7fdc080adde0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fdc040f1d50 con 0x7fdc1c1022a0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc1a59c640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdc080a2c10 0x7fdc080a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009b60 tx=0x7fdc0c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.511+0000 7fdc1ad9d640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] conn(0x7fdc080adde0 0x7fdc080b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.512+0000 7fdc1ad9d640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] conn(0x7fdc080adde0 0x7fdc080b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:02.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.513+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== osd.10 v2:172.21.15.123:6824/3299707057 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc04049e50 con 0x7fdc080adde0 2024-08-19T01:32:02.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.526+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 --> [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc1c06a650 con 0x7fdc080adde0 2024-08-19T01:32:02.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc137fe640 1 -- 172.21.15.64:0/2398783933 <== osd.10 v2:172.21.15.123:6824/3299707057 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdc1c06a650 con 0x7fdc080adde0 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] conn(0x7fdc080adde0 msgr2=0x7fdc080b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] conn(0x7fdc080adde0 0x7fdc080b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdc080a2c10 msgr2=0x7fdc080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fdc080a2c10 0x7fdc080a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009b60 tx=0x7fdc0c005f90 comp rx=0 tx=0).stop 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 msgr2=0x7fdc1c197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 --2- 172.21.15.64:0/2398783933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc1c1022a0 0x7fdc1c197670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdc04006010 tx=0x7fdc04003b90 comp rx=0 tx=0).stop 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc1ad9d640 1 -- 172.21.15.64:0/2398783933 reap_dead start 2024-08-19T01:32:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 shutdown_connections 2024-08-19T01:32:02.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.527+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 >> 172.21.15.64:0/2398783933 conn(0x7fdc1c0fc500 msgr2=0x7fdc1c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:02.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.528+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 shutdown_connections 2024-08-19T01:32:02.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.528+0000 7fdc20a35640 1 -- 172.21.15.64:0/2398783933 wait complete. 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 -- 172.21.15.64:0/1472278973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a78100b50 msgr2=0x7f3a78100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 --2- 172.21.15.64:0/1472278973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a78100b50 0x7f3a78100f30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009ca0 tx=0x7f3a6802a950 comp rx=0 tx=0).stop 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 -- 172.21.15.64:0/1472278973 shutdown_connections 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 --2- 172.21.15.64:0/1472278973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a78102c20 0x7f3a7810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 --2- 172.21.15.64:0/1472278973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a781022a0 0x7f3a781026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 --2- 172.21.15.64:0/1472278973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a78100b50 0x7f3a78100f30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 -- 172.21.15.64:0/1472278973 >> 172.21.15.64:0/1472278973 conn(0x7f3a780fc500 msgr2=0x7f3a780fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 -- 172.21.15.64:0/1472278973 shutdown_connections 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.378+0000 7f3a8035a640 1 -- 172.21.15.64:0/1472278973 wait complete. 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 Processor -- start 2024-08-19T01:32:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 -- start start 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a78100b50 0x7f3a78198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a781022a0 0x7f3a78198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a78102c20 0x7f3a7819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78110cb0 con 0x7f3a781022a0 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78110b30 con 0x7f3a78100b50 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a8035a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78110e30 con 0x7f3a78102c20 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a781022a0 0x7f3a78198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a781022a0 0x7f3a78198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42340/0 (socket says 172.21.15.64:42340) 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 -- 172.21.15.64:0/426456773 learned_addr learned my addr 172.21.15.64:0/426456773 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7e0f6640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a78100b50 0x7f3a78198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7e8f7640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a78102c20 0x7f3a7819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 -- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a78102c20 msgr2=0x7f3a7819cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a78102c20 0x7f3a7819cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 -- 172.21.15.64:0/426456773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a78100b50 msgr2=0x7f3a78198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a78100b50 0x7f3a78198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.379+0000 7f3a7d8f5640 1 -- 172.21.15.64:0/426456773 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7819d1e0 con 0x7f3a781022a0 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.380+0000 7f3a7d8f5640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a781022a0 0x7f3a78198770 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d770 tx=0x7f3a6c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.380+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c017070 con 0x7f3a781022a0 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.380+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c005500 con 0x7f3a781022a0 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.380+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7819d4d0 con 0x7f3a781022a0 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.380+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c004e00 con 0x7f3a781022a0 2024-08-19T01:32:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.380+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a781a4db0 con 0x7f3a781022a0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.381+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3a78069ff0 con 0x7f3a781022a0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.382+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3a6c004fa0 con 0x7f3a781022a0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a777fe640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3a600a2c90 0x7f3a600a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a7e0f6640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3a600a2c90 0x7f3a600a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3a6c0cf310 con 0x7f3a781022a0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a777fe640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] conn(0x7f3a600ade60 0x7f3a600b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 --> [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a6c001d30 con 0x7f3a600ade60 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3a6c0cf760 con 0x7f3a781022a0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a7e0f6640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3a600a2c90 0x7f3a600a5150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3a68004ad0 tx=0x7f3a68004a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.383+0000 7f3a7e8f7640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] conn(0x7f3a600ade60 0x7f3a600b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.384+0000 7f3a7e8f7640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] conn(0x7f3a600ade60 0x7f3a600b0300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.385+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== osd.8 v2:172.21.15.161:6816/2639499139 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3a6c001d30 con 0x7f3a600ade60 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.398+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 --> [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3a78101ee0 con 0x7f3a600ade60 2024-08-19T01:32:02.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a777fe640 1 -- 172.21.15.64:0/426456773 <== osd.8 v2:172.21.15.161:6816/2639499139 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3a78101ee0 con 0x7f3a600ade60 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] conn(0x7f3a600ade60 msgr2=0x7f3a600b0300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] conn(0x7f3a600ade60 0x7f3a600b0300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3a600a2c90 msgr2=0x7f3a600a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f3a600a2c90 0x7f3a600a5150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3a68004ad0 tx=0x7f3a68004a60 comp rx=0 tx=0).stop 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a781022a0 msgr2=0x7f3a78198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 --2- 172.21.15.64:0/426456773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a781022a0 0x7f3a78198770 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d770 tx=0x7f3a6c00dc30 comp rx=0 tx=0).stop 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a7e8f7640 1 -- 172.21.15.64:0/426456773 reap_dead start 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 shutdown_connections 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 >> 172.21.15.64:0/426456773 conn(0x7f3a780fc500 msgr2=0x7f3a78105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.399+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 shutdown_connections 2024-08-19T01:32:02.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.400+0000 7f3a8035a640 1 -- 172.21.15.64:0/426456773 wait complete. 2024-08-19T01:32:03.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 -- 172.21.15.64:0/4083604393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e01022a0 msgr2=0x7f54e01026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 --2- 172.21.15.64:0/4083604393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e01022a0 0x7f54e01026e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009900 tx=0x7f54d402a480 comp rx=0 tx=0).stop 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 -- 172.21.15.64:0/4083604393 shutdown_connections 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 --2- 172.21.15.64:0/4083604393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54e0102c20 0x7f54e010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 --2- 172.21.15.64:0/4083604393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e01022a0 0x7f54e01026e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 --2- 172.21.15.64:0/4083604393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54e0100b50 0x7f54e0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 -- 172.21.15.64:0/4083604393 >> 172.21.15.64:0/4083604393 conn(0x7f54e00fc500 msgr2=0x7f54e00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 -- 172.21.15.64:0/4083604393 shutdown_connections 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 -- 172.21.15.64:0/4083604393 wait complete. 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.312+0000 7f54e7553640 1 Processor -- start 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e7553640 1 -- start start 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e7553640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e0100b50 0x7f54e0198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e7553640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54e01022a0 0x7f54e0199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e7553640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54e0102c20 0x7f54e01a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e7553640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f54e0110cb0 con 0x7f54e0100b50 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e7553640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f54e0110b30 con 0x7f54e0102c20 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e7553640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f54e0110e30 con 0x7f54e01022a0 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e0100b50 0x7f54e0198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e0100b50 0x7f54e0198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42400/0 (socket says 172.21.15.64:42400) 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 -- 172.21.15.64:0/285476 learned_addr learned my addr 172.21.15.64:0/285476 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e5af0640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54e0102c20 0x7f54e01a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e4aee640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54e01022a0 0x7f54e0199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 -- 172.21.15.64:0/285476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54e01022a0 msgr2=0x7f54e0199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54e01022a0 0x7f54e0199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 -- 172.21.15.64:0/285476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54e0102c20 msgr2=0x7f54e01a4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54e0102c20 0x7f54e01a4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e52ef640 1 -- 172.21.15.64:0/285476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e01a45b0 con 0x7f54e0100b50 2024-08-19T01:32:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.313+0000 7f54e5af0640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54e0102c20 0x7f54e01a4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.314+0000 7f54e52ef640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e0100b50 0x7f54e0198ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f54dc00d770 tx=0x7f54dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.314+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc017070 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.314+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54dc005500 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.314+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e01a4780 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.314+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc004e40 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.314+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e01a4c40 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.315+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f54e0107030 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.316+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f54dc009c30 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54d27fc640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f54cc0a2c10 0x7f54cc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54e4aee640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f54cc0a2c10 0x7f54cc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54dc0cf480 con 0x7f54e0100b50 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54d27fc640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] conn(0x7f54cc0adde0 0x7f54cc0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 --> [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54dc016ec0 con 0x7f54cc0adde0 2024-08-19T01:32:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f54dc0cf8d0 con 0x7f54e0100b50 2024-08-19T01:32:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54e5af0640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] conn(0x7f54cc0adde0 0x7f54cc0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54e4aee640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f54cc0a2c10 0x7f54cc0a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f54e006a7f0 tx=0x7f54d4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.317+0000 7f54e5af0640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] conn(0x7f54cc0adde0 0x7f54cc0b0280 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.318+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== osd.3 v2:172.21.15.64:6810/18842482 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54dc016ec0 con 0x7f54cc0adde0 2024-08-19T01:32:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.332+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 --> [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f54e0101ee0 con 0x7f54cc0adde0 2024-08-19T01:32:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.332+0000 7f54d27fc640 1 -- 172.21.15.64:0/285476 <== osd.3 v2:172.21.15.64:6810/18842482 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f54e0101ee0 con 0x7f54cc0adde0 2024-08-19T01:32:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.332+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] conn(0x7f54cc0adde0 msgr2=0x7f54cc0b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.332+0000 7f54e7553640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] conn(0x7f54cc0adde0 0x7f54cc0b0280 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f54cc0a2c10 msgr2=0x7f54cc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f54cc0a2c10 0x7f54cc0a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f54e006a7f0 tx=0x7f54d4005a20 comp rx=0 tx=0).stop 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e0100b50 msgr2=0x7f54e0198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 --2- 172.21.15.64:0/285476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54e0100b50 0x7f54e0198ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f54dc00d770 tx=0x7f54dc00dc30 comp rx=0 tx=0).stop 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e5af0640 1 -- 172.21.15.64:0/285476 reap_dead start 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 shutdown_connections 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 >> 172.21.15.64:0/285476 conn(0x7f54e00fc500 msgr2=0x7f54e00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 shutdown_connections 2024-08-19T01:32:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.333+0000 7f54e7553640 1 -- 172.21.15.64:0/285476 wait complete. 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 -- 172.21.15.64:0/72426107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac8102250 msgr2=0x7fcac810e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 --2- 172.21.15.64:0/72426107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac8102250 0x7fcac810e780 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009cd0 tx=0x7fcac002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 -- 172.21.15.64:0/72426107 shutdown_connections 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 --2- 172.21.15.64:0/72426107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac8102250 0x7fcac810e780 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 --2- 172.21.15.64:0/72426107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac81018b0 0x7fcac8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 --2- 172.21.15.64:0/72426107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac81078b0 0x7fcac8107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 -- 172.21.15.64:0/72426107 >> 172.21.15.64:0/72426107 conn(0x7fcac80fd5f0 msgr2=0x7fcac80ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 -- 172.21.15.64:0/72426107 shutdown_connections 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.794+0000 7fcace91a640 1 -- 172.21.15.64:0/72426107 wait complete. 2024-08-19T01:32:03.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 Processor -- start 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 -- start start 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac81018b0 0x7fcac8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac8102250 0x7fcac8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac81078b0 0x7fcac819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcac8111cb0 con 0x7fcac81078b0 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcac8111b30 con 0x7fcac81018b0 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcace91a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcac8111e30 con 0x7fcac8102250 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcacceb7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac81078b0 0x7fcac819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcacceb7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac81078b0 0x7fcac819ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42420/0 (socket says 172.21.15.64:42420) 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.795+0000 7fcacceb7640 1 -- 172.21.15.64:0/1039740388 learned_addr learned my addr 172.21.15.64:0/1039740388 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcac7fff640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac81018b0 0x7fcac8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcac77fe640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac8102250 0x7fcac8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcacceb7640 1 -- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac8102250 msgr2=0x7fcac8198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcacceb7640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac8102250 0x7fcac8198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcacceb7640 1 -- 172.21.15.64:0/1039740388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac81018b0 msgr2=0x7fcac8198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcacceb7640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac81018b0 0x7fcac8198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcacceb7640 1 -- 172.21.15.64:0/1039740388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac819d0e0 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcac77fe640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac8102250 0x7fcac8198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcacceb7640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac81078b0 0x7fcac819ca00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcac0005cc0 tx=0x7fcac0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0004040 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac00046c0 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac819d370 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0046480 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.796+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac81a4cb0 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.797+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcac80626b0 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.798+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcac00305a0 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.799+0000 7fcac57fa640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fcab00a2c10 0x7fcab00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.799+0000 7fcac7fff640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fcab00a2c10 0x7fcab00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.799+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcac002e030 con 0x7fcac81078b0 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.799+0000 7fcac57fa640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] conn(0x7fcab00adde0 0x7fcab00b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:03.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.799+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 --> [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcab00b0950 con 0x7fcab00adde0 2024-08-19T01:32:03.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.799+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fcac00f1260 con 0x7fcac81078b0 2024-08-19T01:32:03.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.799+0000 7fcac7fff640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fcab00a2c10 0x7fcab00a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcab8005e00 tx=0x7fcab8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:03.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.800+0000 7fcac77fe640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] conn(0x7fcab00adde0 0x7fcab00b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:03.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.800+0000 7fcac77fe640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] conn(0x7fcab00adde0 0x7fcab00b0280 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:03.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.801+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== osd.2 v2:172.21.15.161:6800/2169170995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcab00b0950 con 0x7fcab00adde0 2024-08-19T01:32:03.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:03 smithi064 ceph-mon[18515]: pgmap v305: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 767 B/s wr, 6 op/s 2024-08-19T01:32:03.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.815+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 --> [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcac8062b30 con 0x7fcab00adde0 2024-08-19T01:32:03.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcac57fa640 1 -- 172.21.15.64:0/1039740388 <== osd.2 v2:172.21.15.161:6800/2169170995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcac8062b30 con 0x7fcab00adde0 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] conn(0x7fcab00adde0 msgr2=0x7fcab00b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] conn(0x7fcab00adde0 0x7fcab00b0280 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fcab00a2c10 msgr2=0x7fcab00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fcab00a2c10 0x7fcab00a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcab8005e00 tx=0x7fcab8005c50 comp rx=0 tx=0).stop 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac81078b0 msgr2=0x7fcac819ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 --2- 172.21.15.64:0/1039740388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcac81078b0 0x7fcac819ca00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcac0005cc0 tx=0x7fcac0003970 comp rx=0 tx=0).stop 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcacceb7640 1 -- 172.21.15.64:0/1039740388 reap_dead start 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 shutdown_connections 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 >> 172.21.15.64:0/1039740388 conn(0x7fcac80fd5f0 msgr2=0x7fcac80fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 shutdown_connections 2024-08-19T01:32:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:03.816+0000 7fcace91a640 1 -- 172.21.15.64:0/1039740388 wait complete. 2024-08-19T01:32:03.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:03 smithi123 ceph-mon[21829]: pgmap v305: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 767 B/s wr, 6 op/s 2024-08-19T01:32:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:03 smithi161 ceph-mon[21757]: pgmap v305: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 767 B/s wr, 6 op/s 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 -- 172.21.15.64:0/3549377542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a281078b0 msgr2=0x7f9a28107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 --2- 172.21.15.64:0/3549377542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a281078b0 0x7f9a28107c90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009930 tx=0x7f9a1802a420 comp rx=0 tx=0).stop 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 -- 172.21.15.64:0/3549377542 shutdown_connections 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 --2- 172.21.15.64:0/3549377542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a28102250 0x7f9a2810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 --2- 172.21.15.64:0/3549377542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a281018b0 0x7f9a28101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 --2- 172.21.15.64:0/3549377542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a281078b0 0x7f9a28107c90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 -- 172.21.15.64:0/3549377542 >> 172.21.15.64:0/3549377542 conn(0x7f9a280fd5f0 msgr2=0x7f9a280ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 -- 172.21.15.64:0/3549377542 shutdown_connections 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.021+0000 7f9a2d408640 1 -- 172.21.15.64:0/3549377542 wait complete. 2024-08-19T01:32:04.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 Processor -- start 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 -- start start 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a281018b0 0x7f9a28198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a28102250 0x7f9a28198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a281078b0 0x7f9a2819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a28111cf0 con 0x7f9a28102250 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9a28111b70 con 0x7f9a281018b0 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a2d408640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9a28111e70 con 0x7f9a281078b0 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a28102250 0x7f9a28198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a28102250 0x7f9a28198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42428/0 (socket says 172.21.15.64:42428) 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 -- 172.21.15.64:0/2512303546 learned_addr learned my addr 172.21.15.64:0/2512303546 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a277fe640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a281078b0 0x7f9a2819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 -- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a281078b0 msgr2=0x7f9a2819cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a281078b0 0x7f9a2819cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a26ffd640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a281018b0 0x7f9a28198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 -- 172.21.15.64:0/2512303546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a281018b0 msgr2=0x7f9a28198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a281018b0 0x7f9a28198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a1e7fc640 1 -- 172.21.15.64:0/2512303546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2819d200 con 0x7f9a28102250 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.022+0000 7f9a277fe640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a281078b0 0x7f9a2819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.023+0000 7f9a1e7fc640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a28102250 0x7f9a28198790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000d770 tx=0x7f9a1000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.023+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10017070 con 0x7f9a28102250 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.023+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a10005500 con 0x7f9a28102250 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.023+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2819d4f0 con 0x7f9a28102250 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.023+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10004e00 con 0x7f9a28102250 2024-08-19T01:32:04.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.023+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a281a4dd0 con 0x7f9a28102250 2024-08-19T01:32:04.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.024+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9a280626b0 con 0x7f9a28102250 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.025+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9a10004fa0 con 0x7f9a28102250 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.026+0000 7f9a24ff9640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a140a2c10 0x7f9a140a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.026+0000 7f9a26ffd640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a140a2c10 0x7f9a140a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.026+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9a100cf450 con 0x7f9a28102250 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.026+0000 7f9a24ff9640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] conn(0x7f9a140adde0 0x7f9a140b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.026+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 --> [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a10001d30 con 0x7f9a140adde0 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.026+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9a100cf8a0 con 0x7f9a28102250 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.026+0000 7f9a26ffd640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a140a2c10 0x7f9a140a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a180096c0 tx=0x7f9a18000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.027+0000 7f9a277fe640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] conn(0x7f9a140adde0 0x7f9a140b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.027+0000 7f9a277fe640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] conn(0x7f9a140adde0 0x7f9a140b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:04.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.028+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== osd.5 v2:172.21.15.161:6808/3335610027 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a10001d30 con 0x7f9a140adde0 2024-08-19T01:32:04.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.042+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 --> [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9a28062b30 con 0x7f9a140adde0 2024-08-19T01:32:04.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.042+0000 7f9a24ff9640 1 -- 172.21.15.64:0/2512303546 <== osd.5 v2:172.21.15.161:6808/3335610027 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9a28062b30 con 0x7f9a140adde0 2024-08-19T01:32:04.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.042+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] conn(0x7f9a140adde0 msgr2=0x7f9a140b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.042+0000 7f9a2d408640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] conn(0x7f9a140adde0 0x7f9a140b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a140a2c10 msgr2=0x7f9a140a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a140a2c10 0x7f9a140a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a180096c0 tx=0x7f9a18000bc0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a28102250 msgr2=0x7f9a28198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 --2- 172.21.15.64:0/2512303546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a28102250 0x7f9a28198790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000d770 tx=0x7f9a1000dc30 comp rx=0 tx=0).stop 2024-08-19T01:32:04.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a277fe640 1 -- 172.21.15.64:0/2512303546 reap_dead start 2024-08-19T01:32:04.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 shutdown_connections 2024-08-19T01:32:04.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 >> 172.21.15.64:0/2512303546 conn(0x7f9a280fd5f0 msgr2=0x7f9a280fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:04.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 shutdown_connections 2024-08-19T01:32:04.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.043+0000 7f9a2d408640 1 -- 172.21.15.64:0/2512303546 wait complete. 2024-08-19T01:32:04.588 INFO:teuthology.orchestra.run.smithi064.stdout:103079215167 2024-08-19T01:32:04.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-19T01:32:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 -- 172.21.15.64:0/3234122869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba14100b50 msgr2=0x7fba14100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 --2- 172.21.15.64:0/3234122869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba14100b50 0x7fba14100f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fba04009900 tx=0x7fba0402a480 comp rx=0 tx=0).stop 2024-08-19T01:32:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 -- 172.21.15.64:0/3234122869 shutdown_connections 2024-08-19T01:32:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 --2- 172.21.15.64:0/3234122869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba14102c20 0x7fba1410d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 --2- 172.21.15.64:0/3234122869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba141022a0 0x7fba141026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 --2- 172.21.15.64:0/3234122869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba14100b50 0x7fba14100f30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 -- 172.21.15.64:0/3234122869 >> 172.21.15.64:0/3234122869 conn(0x7fba140fc500 msgr2=0x7fba140fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 -- 172.21.15.64:0/3234122869 shutdown_connections 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.766+0000 7fba1a441640 1 -- 172.21.15.64:0/3234122869 wait complete. 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 Processor -- start 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 -- start start 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba14100b50 0x7fba14197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba141022a0 0x7fba141976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba14102c20 0x7fba1419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fba14110cf0 con 0x7fba141022a0 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fba14110b70 con 0x7fba14102c20 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba1a441640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fba14110e70 con 0x7fba14100b50 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba0bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba141022a0 0x7fba141976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba0bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba141022a0 0x7fba141976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42460/0 (socket says 172.21.15.64:42460) 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba0bfff640 1 -- 172.21.15.64:0/3063705299 learned_addr learned my addr 172.21.15.64:0/3063705299 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba13fff640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba14100b50 0x7fba14197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.767+0000 7fba189de640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba14102c20 0x7fba1419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba0bfff640 1 -- 172.21.15.64:0/3063705299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba14100b50 msgr2=0x7fba14197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba0bfff640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba14100b50 0x7fba14197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba0bfff640 1 -- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba14102c20 msgr2=0x7fba1419ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba0bfff640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba14102c20 0x7fba1419ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba0bfff640 1 -- 172.21.15.64:0/3063705299 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1419c140 con 0x7fba141022a0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba0bfff640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba141022a0 0x7fba141976d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d770 tx=0x7fb9fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc017070 con 0x7fba141022a0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9fc005500 con 0x7fba141022a0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc004e00 con 0x7fba141022a0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1419c430 con 0x7fba141022a0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.768+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba141a3d10 con 0x7fba141022a0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.769+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba140626b0 con 0x7fba141022a0 2024-08-19T01:32:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.770+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb9fc004fa0 con 0x7fba141022a0 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.771+0000 7fba11ffb640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fba000a2c90 0x7fba000a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.771+0000 7fba13fff640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fba000a2c90 0x7fba000a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.771+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb9fc0cf310 con 0x7fba141022a0 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.771+0000 7fba11ffb640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] conn(0x7fba000ade60 0x7fba000b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.771+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 --> [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9fc001d30 con 0x7fba000ade60 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.771+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb9fc0cf760 con 0x7fba141022a0 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.772+0000 7fba13fff640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fba000a2c90 0x7fba000a5150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba04009230 tx=0x7fba040058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.772+0000 7fba189de640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] conn(0x7fba000ade60 0x7fba000b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.772+0000 7fba189de640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] conn(0x7fba000ade60 0x7fba000b0300 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.773+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== osd.7 v2:172.21.15.123:6816/2282165351 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9fc001d30 con 0x7fba000ade60 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.787+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 --> [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fba14062b30 con 0x7fba000ade60 2024-08-19T01:32:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.788+0000 7fba11ffb640 1 -- 172.21.15.64:0/3063705299 <== osd.7 v2:172.21.15.123:6816/2282165351 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fba14062b30 con 0x7fba000ade60 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.788+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] conn(0x7fba000ade60 msgr2=0x7fba000b0300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.788+0000 7fba1a441640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] conn(0x7fba000ade60 0x7fba000b0300 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.788+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fba000a2c90 msgr2=0x7fba000a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.788+0000 7fba1a441640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fba000a2c90 0x7fba000a5150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba04009230 tx=0x7fba040058d0 comp rx=0 tx=0).stop 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.789+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba141022a0 msgr2=0x7fba141976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.789+0000 7fba1a441640 1 --2- 172.21.15.64:0/3063705299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba141022a0 0x7fba141976d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d770 tx=0x7fb9fc00dc30 comp rx=0 tx=0).stop 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.789+0000 7fba189de640 1 -- 172.21.15.64:0/3063705299 reap_dead start 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.789+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 shutdown_connections 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.789+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 >> 172.21.15.64:0/3063705299 conn(0x7fba140fc500 msgr2=0x7fba140fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.789+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 shutdown_connections 2024-08-19T01:32:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:04.789+0000 7fba1a441640 1 -- 172.21.15.64:0/3063705299 wait complete. 2024-08-19T01:32:05.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:05 smithi064 ceph-mon[18515]: pgmap v306: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 -- 172.21.15.64:0/3798400743 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 msgr2=0x7f059410d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 --2- 172.21.15.64:0/3798400743 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 0x7f059410d690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f058c009cd0 tx=0x7f058c02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 -- 172.21.15.64:0/3798400743 shutdown_connections 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 --2- 172.21.15.64:0/3798400743 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 0x7f059410d690 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 --2- 172.21.15.64:0/3798400743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05941007c0 0x7f0594100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 --2- 172.21.15.64:0/3798400743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05941067c0 0x7f0594106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 -- 172.21.15.64:0/3798400743 >> 172.21.15.64:0/3798400743 conn(0x7f05940fc500 msgr2=0x7f05940fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.741+0000 7f059a4f3640 1 -- 172.21.15.64:0/3798400743 shutdown_connections 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 -- 172.21.15.64:0/3798400743 wait complete. 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 Processor -- start 2024-08-19T01:32:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 -- start start 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05941007c0 0x7f0594198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 0x7f0594198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05941067c0 0x7f059419c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0594110c10 con 0x7f0594101160 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0594110a90 con 0x7f05941007c0 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f059a4f3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0594110d90 con 0x7f05941067c0 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f058bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 0x7f0594198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f058bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 0x7f0594198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42482/0 (socket says 172.21.15.64:42482) 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.742+0000 7f058bfff640 1 -- 172.21.15.64:0/1950269858 learned_addr learned my addr 172.21.15.64:0/1950269858 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f0593fff640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05941007c0 0x7f0594198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f0598a90640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05941067c0 0x7f059419c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f058bfff640 1 -- 172.21.15.64:0/1950269858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05941067c0 msgr2=0x7f059419c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f058bfff640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05941067c0 0x7f059419c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f058bfff640 1 -- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05941007c0 msgr2=0x7f0594198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f058bfff640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05941007c0 0x7f0594198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f058bfff640 1 -- 172.21.15.64:0/1950269858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059419d0d0 con 0x7f0594101160 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f058bfff640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 0x7f0594198660 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f057c00e9f0 tx=0x7f057c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c00cd80 con 0x7f0594101160 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f057c010910 con 0x7f0594101160 2024-08-19T01:32:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059419d3c0 con 0x7f0594101160 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c018990 con 0x7f0594101160 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.743+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05941a4ca0 con 0x7f0594101160 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.744+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f059406b000 con 0x7f0594101160 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.745+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f057c010ac0 con 0x7f0594101160 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0591ffb640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f05800a2c10 0x7f05800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0593fff640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f05800a2c10 0x7f05800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f057c0d0010 con 0x7f0594101160 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0591ffb640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] conn(0x7f05800adde0 0x7f05800b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 --> [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f057c022e50 con 0x7f05800adde0 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f057c0d0460 con 0x7f0594101160 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0593fff640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f05800a2c10 0x7f05800a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0584009ae0 tx=0x7f0584005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.746+0000 7f0598a90640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] conn(0x7f05800adde0 0x7f05800b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.747+0000 7f0598a90640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] conn(0x7f05800adde0 0x7f05800b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.748+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== osd.1 v2:172.21.15.123:6800/4102727954 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f057c022e50 con 0x7f05800adde0 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.761+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 --> [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0594101f00 con 0x7f05800adde0 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f0591ffb640 1 -- 172.21.15.64:0/1950269858 <== osd.1 v2:172.21.15.123:6800/4102727954 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0594101f00 con 0x7f05800adde0 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] conn(0x7f05800adde0 msgr2=0x7f05800b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] conn(0x7f05800adde0 0x7f05800b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f05800a2c10 msgr2=0x7f05800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f05800a2c10 0x7f05800a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0584009ae0 tx=0x7f0584005f90 comp rx=0 tx=0).stop 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 msgr2=0x7f0594198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 --2- 172.21.15.64:0/1950269858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0594101160 0x7f0594198660 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f057c00e9f0 tx=0x7f057c00eeb0 comp rx=0 tx=0).stop 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f0598a90640 1 -- 172.21.15.64:0/1950269858 reap_dead start 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 shutdown_connections 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 >> 172.21.15.64:0/1950269858 conn(0x7f05940fc500 msgr2=0x7f059406a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 shutdown_connections 2024-08-19T01:32:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:05.762+0000 7f059a4f3640 1 -- 172.21.15.64:0/1950269858 wait complete. 2024-08-19T01:32:05.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:05 smithi123 ceph-mon[21829]: pgmap v306: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-08-19T01:32:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:05 smithi161 ceph-mon[21757]: pgmap v306: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-08-19T01:32:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 -- 172.21.15.64:0/2044442482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 msgr2=0x7f441410d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 --2- 172.21.15.64:0/2044442482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 0x7f441410d700 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f440c009cd0 tx=0x7f440c02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 -- 172.21.15.64:0/2044442482 shutdown_connections 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 --2- 172.21.15.64:0/2044442482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 0x7f441410d700 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 --2- 172.21.15.64:0/2044442482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4414101ae0 0x7f44141053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 --2- 172.21.15.64:0/2044442482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4414101130 0x7f4414101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 -- 172.21.15.64:0/2044442482 >> 172.21.15.64:0/2044442482 conn(0x7f44140fbc10 msgr2=0x7f44140fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 -- 172.21.15.64:0/2044442482 shutdown_connections 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.303+0000 7f4418ad6640 1 -- 172.21.15.64:0/2044442482 wait complete. 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 Processor -- start 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 -- start start 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4414101130 0x7f4414198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4414101ae0 0x7f44141986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 0x7f441419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4414110bb0 con 0x7f4414105b20 2024-08-19T01:32:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4414110a30 con 0x7f4414101130 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4418ad6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4414110d30 con 0x7f4414101ae0 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4412d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 0x7f441419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4412d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 0x7f441419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42502/0 (socket says 172.21.15.64:42502) 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.304+0000 7f4412d9d640 1 -- 172.21.15.64:0/3741665796 learned_addr learned my addr 172.21.15.64:0/3741665796 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f441259c640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4414101130 0x7f4414198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4411d9b640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4414101ae0 0x7f44141986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4412d9d640 1 -- 172.21.15.64:0/3741665796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4414101ae0 msgr2=0x7f44141986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4412d9d640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4414101ae0 0x7f44141986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4412d9d640 1 -- 172.21.15.64:0/3741665796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4414101130 msgr2=0x7f4414198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4412d9d640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4414101130 0x7f4414198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4412d9d640 1 -- 172.21.15.64:0/3741665796 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441419d130 con 0x7f4414105b20 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4412d9d640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 0x7f441419ca50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f440c005cc0 tx=0x7f440c0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c003e00 con 0x7f4414105b20 2024-08-19T01:32:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f440c004520 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f441419d3c0 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c030db0 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.305+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44141a4d00 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.306+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4414104860 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.307+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f440c0046d0 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.308+0000 7f44037fe640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f43fc0a2c10 0x7f43fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.308+0000 7f441259c640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f43fc0a2c10 0x7f43fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.308+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f440c02e030 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.308+0000 7f44037fe640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] conn(0x7f43fc0adde0 0x7f43fc0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.308+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 --> [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f440c0020b0 con 0x7f43fc0adde0 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.308+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f440c0f1120 con 0x7f4414105b20 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.309+0000 7f4411d9b640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] conn(0x7f43fc0adde0 0x7f43fc0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.309+0000 7f441259c640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f43fc0a2c10 0x7f43fc0a50d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4404009b60 tx=0x7f4404005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.309+0000 7f4411d9b640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] conn(0x7f43fc0adde0 0x7f43fc0b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.310+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== osd.6 v2:172.21.15.64:6818/3144814721 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f440c0020b0 con 0x7f43fc0adde0 2024-08-19T01:32:06.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.323+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 --> [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f44141020f0 con 0x7f43fc0adde0 2024-08-19T01:32:06.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f44037fe640 1 -- 172.21.15.64:0/3741665796 <== osd.6 v2:172.21.15.64:6818/3144814721 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f44141020f0 con 0x7f43fc0adde0 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] conn(0x7f43fc0adde0 msgr2=0x7f43fc0b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] conn(0x7f43fc0adde0 0x7f43fc0b0280 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f43fc0a2c10 msgr2=0x7f43fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f43fc0a2c10 0x7f43fc0a50d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4404009b60 tx=0x7f4404005f90 comp rx=0 tx=0).stop 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 msgr2=0x7f441419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 --2- 172.21.15.64:0/3741665796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4414105b20 0x7f441419ca50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f440c005cc0 tx=0x7f440c0093d0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4412d9d640 1 -- 172.21.15.64:0/3741665796 reap_dead start 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 shutdown_connections 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 >> 172.21.15.64:0/3741665796 conn(0x7f44140fbc10 msgr2=0x7f44140fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 shutdown_connections 2024-08-19T01:32:06.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.324+0000 7f4418ad6640 1 -- 172.21.15.64:0/3741665796 wait complete. 2024-08-19T01:32:06.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 -- 172.21.15.64:0/3061636959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 msgr2=0x7f5a14106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 --2- 172.21.15.64:0/3061636959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 0x7f5a14106b90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009930 tx=0x7f5a0402a420 comp rx=0 tx=0).stop 2024-08-19T01:32:06.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 -- 172.21.15.64:0/3061636959 shutdown_connections 2024-08-19T01:32:06.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 --2- 172.21.15.64:0/3061636959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a14101150 0x7f5a1410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 --2- 172.21.15.64:0/3061636959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a141007b0 0x7f5a14100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 --2- 172.21.15.64:0/3061636959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 0x7f5a14106b90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 -- 172.21.15.64:0/3061636959 >> 172.21.15.64:0/3061636959 conn(0x7f5a140fc4d0 msgr2=0x7f5a140fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.693+0000 7f5a1a753640 1 -- 172.21.15.64:0/3061636959 shutdown_connections 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.694+0000 7f5a1a753640 1 -- 172.21.15.64:0/3061636959 wait complete. 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.694+0000 7f5a1a753640 1 Processor -- start 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.694+0000 7f5a1a753640 1 -- start start 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.694+0000 7f5a1a753640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a141007b0 0x7f5a141981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a1a753640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a14101150 0x7f5a14198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a1a753640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 0x7f5a1419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a1a753640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5a14110bd0 con 0x7f5a141067b0 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a1a753640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5a14110a50 con 0x7f5a14101150 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a1a753640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a14110d50 con 0x7f5a141007b0 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 0x7f5a1419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 0x7f5a1419ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42372/0 (socket says 172.21.15.64:42372) 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 -- 172.21.15.64:0/1727809848 learned_addr learned my addr 172.21.15.64:0/1727809848 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a13fff640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a141007b0 0x7f5a141981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a137fe640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a14101150 0x7f5a14198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 -- 172.21.15.64:0/1727809848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a141007b0 msgr2=0x7f5a141981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a141007b0 0x7f5a141981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 -- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a14101150 msgr2=0x7f5a14198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a14101150 0x7f5a14198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 -- 172.21.15.64:0/1727809848 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1419d170 con 0x7f5a141067b0 2024-08-19T01:32:06.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a18cf0640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 0x7f5a1419ca90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00b970 tx=0x7f5a0c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c019070 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1419d460 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0c00cb40 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c007610 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.695+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a141a4d40 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.696+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a1406b060 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.698+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5a0c01c070 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.698+0000 7f5a117fa640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f59fc0a2c10 0x7f59fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.698+0000 7f5a13fff640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f59fc0a2c10 0x7f59fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.698+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a0c0cf200 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.698+0000 7f5a117fa640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] conn(0x7f59fc0adde0 0x7f59fc0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.698+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 --> [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a0c0047d0 con 0x7f59fc0adde0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.698+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f5a0c0cf650 con 0x7f5a141067b0 2024-08-19T01:32:06.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.699+0000 7f5a137fe640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] conn(0x7f59fc0adde0 0x7f59fc0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.699+0000 7f5a13fff640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f59fc0a2c10 0x7f59fc0a50d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009900 tx=0x7f5a040058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.699+0000 7f5a137fe640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] conn(0x7f59fc0adde0 0x7f59fc0b0280 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.700+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== osd.4 v2:172.21.15.123:6808/529776515 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a0c0047d0 con 0x7f59fc0adde0 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.713+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 --> [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5a14101ef0 con 0x7f59fc0adde0 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a117fa640 1 -- 172.21.15.64:0/1727809848 <== osd.4 v2:172.21.15.123:6808/529776515 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5a14101ef0 con 0x7f59fc0adde0 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] conn(0x7f59fc0adde0 msgr2=0x7f59fc0b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] conn(0x7f59fc0adde0 0x7f59fc0b0280 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f59fc0a2c10 msgr2=0x7f59fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f59fc0a2c10 0x7f59fc0a50d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009900 tx=0x7f5a040058d0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 msgr2=0x7f5a1419ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 --2- 172.21.15.64:0/1727809848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a141067b0 0x7f5a1419ca90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00b970 tx=0x7f5a0c00be30 comp rx=0 tx=0).stop 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a18cf0640 1 -- 172.21.15.64:0/1727809848 reap_dead start 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 shutdown_connections 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.714+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 >> 172.21.15.64:0/1727809848 conn(0x7f5a140fc4d0 msgr2=0x7f5a1406a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:06.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.715+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 shutdown_connections 2024-08-19T01:32:06.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:02.715+0000 7f5a1a753640 1 -- 172.21.15.64:0/1727809848 wait complete. 2024-08-19T01:32:06.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/55166860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a941033f0 msgr2=0x7f6a9410b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 --2- 172.21.15.64:0/55166860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a941033f0 0x7f6a9410b8e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009900 tx=0x7f6a8402a480 comp rx=0 tx=0).stop 2024-08-19T01:32:06.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/55166860 shutdown_connections 2024-08-19T01:32:06.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 --2- 172.21.15.64:0/55166860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9410beb0 0x7f6a9410e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 --2- 172.21.15.64:0/55166860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a941033f0 0x7f6a9410b8e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 --2- 172.21.15.64:0/55166860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a94102ad0 0x7f6a94102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/55166860 >> 172.21.15.64:0/55166860 conn(0x7f6a940fc4d0 msgr2=0x7f6a940fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/55166860 shutdown_connections 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.194+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/55166860 wait complete. 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 Processor -- start 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 -- start start 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a94102ad0 0x7f6a94198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a941033f0 0x7f6a941985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9410beb0 0x7f6a9419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6a94111c90 con 0x7f6a94102ad0 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a94111b10 con 0x7f6a941033f0 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9bdd2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a94111e10 con 0x7f6a9410beb0 2024-08-19T01:32:06.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a94102ad0 0x7f6a94198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a94102ad0 0x7f6a94198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42490/0 (socket says 172.21.15.64:42490) 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 -- 172.21.15.64:0/614206364 learned_addr learned my addr 172.21.15.64:0/614206364 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9936d640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a941033f0 0x7f6a941985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9a36f640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9410beb0 0x7f6a9419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 -- 172.21.15.64:0/614206364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9410beb0 msgr2=0x7f6a9419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9410beb0 0x7f6a9419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 -- 172.21.15.64:0/614206364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a941033f0 msgr2=0x7f6a941985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a941033f0 0x7f6a941985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a99b6e640 1 -- 172.21.15.64:0/614206364 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9419d030 con 0x7f6a94102ad0 2024-08-19T01:32:06.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9936d640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a941033f0 0x7f6a941985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.195+0000 7f6a9a36f640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a9410beb0 0x7f6a9419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.196+0000 7f6a99b6e640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a94102ad0 0x7f6a94198080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000c920 tx=0x7f6a9000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.196+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90007c70 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.196+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9419d320 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.196+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a90007e10 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.196+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90018890 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.196+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a941a4c00 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.197+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6a940626b0 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.198+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6a90018a50 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a8affd640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a800a2c10 0x7f6a800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a9936d640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a800a2c10 0x7f6a800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6a900cfb20 con 0x7f6a94102ad0 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a8affd640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] conn(0x7f6a800adde0 0x7f6a800b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 --> [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a800b0950 con 0x7f6a800adde0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f6a900cff70 con 0x7f6a94102ad0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a9a36f640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] conn(0x7f6a800adde0 0x7f6a800b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.199+0000 7f6a9936d640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a800a2c10 0x7f6a800a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6a9410d2d0 tx=0x7f6a84005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.200+0000 7f6a9a36f640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] conn(0x7f6a800adde0 0x7f6a800b0280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.200+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== osd.0 v2:172.21.15.64:6802/3320376368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a800b0950 con 0x7f6a800adde0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 --> [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6a94062ab0 con 0x7f6a800adde0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a8affd640 1 -- 172.21.15.64:0/614206364 <== osd.0 v2:172.21.15.64:6802/3320376368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6a94062ab0 con 0x7f6a800adde0 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] conn(0x7f6a800adde0 msgr2=0x7f6a800b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a9bdd2640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] conn(0x7f6a800adde0 0x7f6a800b0280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a800a2c10 msgr2=0x7f6a800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a9bdd2640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f6a800a2c10 0x7f6a800a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6a9410d2d0 tx=0x7f6a84005a20 comp rx=0 tx=0).stop 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a94102ad0 msgr2=0x7f6a94198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.214+0000 7f6a9bdd2640 1 --2- 172.21.15.64:0/614206364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a94102ad0 0x7f6a94198080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000c920 tx=0x7f6a9000cde0 comp rx=0 tx=0).stop 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.215+0000 7f6a9a36f640 1 -- 172.21.15.64:0/614206364 reap_dead start 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.215+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 shutdown_connections 2024-08-19T01:32:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.215+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 >> 172.21.15.64:0/614206364 conn(0x7f6a940fc4d0 msgr2=0x7f6a940fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:06.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.215+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 shutdown_connections 2024-08-19T01:32:06.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:06.215+0000 7f6a9bdd2640 1 -- 172.21.15.64:0/614206364 wait complete. 2024-08-19T01:32:07.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:07 smithi064 ceph-mon[18515]: pgmap v307: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-08-19T01:32:07.824 INFO:teuthology.orchestra.run.smithi064.stdout:133143986236 2024-08-19T01:32:07.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-19T01:32:07.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:07 smithi123 ceph-mon[21829]: pgmap v307: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-08-19T01:32:07.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:07 smithi161 ceph-mon[21757]: pgmap v307: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-08-19T01:32:08.245 INFO:teuthology.orchestra.run.smithi064.stdout:34359738445 2024-08-19T01:32:08.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-19T01:32:08.649 INFO:teuthology.orchestra.run.smithi064.stdout:296352743455 2024-08-19T01:32:08.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-08-19T01:32:09.079 INFO:teuthology.orchestra.run.smithi064.stdout:240518168616 2024-08-19T01:32:09.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-08-19T01:32:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.684+0000 7f1070314640 1 -- 172.21.15.64:0/4083935651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 msgr2=0x7f1068102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.684+0000 7f1070314640 1 --2- 172.21.15.64:0/4083935651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 0x7f1068102ea0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1060009930 tx=0x7f106002a420 comp rx=0 tx=0).stop 2024-08-19T01:32:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- 172.21.15.64:0/4083935651 shutdown_connections 2024-08-19T01:32:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 --2- 172.21.15.64:0/4083935651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1068107fc0 0x7f106810e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 --2- 172.21.15.64:0/4083935651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1068103470 0x7f10681078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 --2- 172.21.15.64:0/4083935651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 0x7f1068102ea0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- 172.21.15.64:0/4083935651 >> 172.21.15.64:0/4083935651 conn(0x7f10680fc500 msgr2=0x7f10680fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- 172.21.15.64:0/4083935651 shutdown_connections 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- 172.21.15.64:0/4083935651 wait complete. 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 Processor -- start 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- start start 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 0x7f1068198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1068103470 0x7f1068198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1068107fc0 0x7f106819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1068111dc0 con 0x7f1068102ac0 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1068111c40 con 0x7f1068107fc0 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.685+0000 7f1070314640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1068111f40 con 0x7f1068103470 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 0x7f1068198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 0x7f1068198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40540/0 (socket says 172.21.15.64:40540) 2024-08-19T01:32:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 -- 172.21.15.64:0/136489640 learned_addr learned my addr 172.21.15.64:0/136489640 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106d8af640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1068103470 0x7f1068198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e8b1640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1068107fc0 0x7f106819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 -- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1068103470 msgr2=0x7f1068198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1068103470 0x7f1068198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 -- 172.21.15.64:0/136489640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1068107fc0 msgr2=0x7f106819cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1068107fc0 0x7f106819cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 -- 172.21.15.64:0/136489640 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106819d2d0 con 0x7f1068102ac0 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f106e0b0640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 0x7f1068198320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1060009230 tx=0x7f1060004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106003f030 con 0x7f1068102ac0 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1060003ad0 con 0x7f1068102ac0 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106819d560 con 0x7f1068102ac0 2024-08-19T01:32:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060035070 con 0x7f1068102ac0 2024-08-19T01:32:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.686+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10681a4ea0 con 0x7f1068102ac0 2024-08-19T01:32:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.687+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10680626b0 con 0x7f1068102ac0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.689+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1060042030 con 0x7f1068102ac0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.689+0000 7f10677fe640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f10500a2c10 0x7f10500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.689+0000 7f106d8af640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f10500a2c10 0x7f10500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.689+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f10600eff00 con 0x7f1068102ac0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.689+0000 7f10677fe640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] conn(0x7f10500adde0 0x7f10500b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.689+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 --> [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10500b0950 con 0x7f10500adde0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.689+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f10600f0350 con 0x7f1068102ac0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.690+0000 7f106e8b1640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] conn(0x7f10500adde0 0x7f10500b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.690+0000 7f106d8af640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f10500a2c10 0x7f10500a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f105c005fd0 tx=0x7f105c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.690+0000 7f106e8b1640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] conn(0x7f10500adde0 0x7f10500b0280 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.691+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== osd.11 v2:172.21.15.161:6824/564834779 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10500b0950 con 0x7f10500adde0 2024-08-19T01:32:09.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.705+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 --> [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1068062b30 con 0x7f10500adde0 2024-08-19T01:32:09.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.705+0000 7f10677fe640 1 -- 172.21.15.64:0/136489640 <== osd.11 v2:172.21.15.161:6824/564834779 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1068062b30 con 0x7f10500adde0 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] conn(0x7f10500adde0 msgr2=0x7f10500b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] conn(0x7f10500adde0 0x7f10500b0280 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f10500a2c10 msgr2=0x7f10500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f10500a2c10 0x7f10500a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f105c005fd0 tx=0x7f105c005850 comp rx=0 tx=0).stop 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 msgr2=0x7f1068198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 --2- 172.21.15.64:0/136489640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1068102ac0 0x7f1068198320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1060009230 tx=0x7f1060004270 comp rx=0 tx=0).stop 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f106e8b1640 1 -- 172.21.15.64:0/136489640 reap_dead start 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 shutdown_connections 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 >> 172.21.15.64:0/136489640 conn(0x7f10680fc500 msgr2=0x7f1068105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 shutdown_connections 2024-08-19T01:32:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.706+0000 7f1070314640 1 -- 172.21.15.64:0/136489640 wait complete. 2024-08-19T01:32:09.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:09 smithi064 ceph-mon[18515]: pgmap v308: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-08-19T01:32:09.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:09 smithi123 ceph-mon[21829]: pgmap v308: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-08-19T01:32:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:09 smithi161 ceph-mon[21757]: pgmap v308: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-08-19T01:32:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.903+0000 7fb1927ae640 1 -- 172.21.15.64:0/2336752648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c106c00 msgr2=0x7fb18c10e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.903+0000 7fb1927ae640 1 --2- 172.21.15.64:0/2336752648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c106c00 0x7fb18c10e7d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb184009ca0 tx=0x7fb18402a8c0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.903+0000 7fb1927ae640 1 -- 172.21.15.64:0/2336752648 shutdown_connections 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.903+0000 7fb1927ae640 1 --2- 172.21.15.64:0/2336752648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c106c00 0x7fb18c10e7d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.903+0000 7fb1927ae640 1 --2- 172.21.15.64:0/2336752648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb18c100be0 0x7fb18c1064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.903+0000 7fb1927ae640 1 --2- 172.21.15.64:0/2336752648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb18c100230 0x7fb18c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.903+0000 7fb1927ae640 1 -- 172.21.15.64:0/2336752648 >> 172.21.15.64:0/2336752648 conn(0x7fb18c0fc2f0 msgr2=0x7fb18c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.904+0000 7fb1927ae640 1 -- 172.21.15.64:0/2336752648 shutdown_connections 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.904+0000 7fb1927ae640 1 -- 172.21.15.64:0/2336752648 wait complete. 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.904+0000 7fb1927ae640 1 Processor -- start 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.904+0000 7fb1927ae640 1 -- start start 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c100230 0x7fb18c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb18c100be0 0x7fb18c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb18c106c00 0x7fb18c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111cb0 con 0x7fb18c100230 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111b30 con 0x7fb18c100be0 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111e30 con 0x7fb18c106c00 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c100230 0x7fb18c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c100230 0x7fb18c1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40554/0 (socket says 172.21.15.64:40554) 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 -- 172.21.15.64:0/366099843 learned_addr learned my addr 172.21.15.64:0/366099843 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb183fff640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb18c100be0 0x7fb18c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb190d4b640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb18c106c00 0x7fb18c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 -- 172.21.15.64:0/366099843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb18c106c00 msgr2=0x7fb18c19c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb18c106c00 0x7fb18c19c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 -- 172.21.15.64:0/366099843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb18c100be0 msgr2=0x7fb18c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb18c100be0 0x7fb18c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 -- 172.21.15.64:0/366099843 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c19d0a0 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb18bfff640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c100230 0x7fb18c1980f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d770 tx=0x7fb17c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c017070 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb17c005500 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c19d390 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c004e00 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.905+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c1a4c70 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.906+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb18c0626b0 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.908+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb17c004fa0 con 0x7fb18c100230 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.908+0000 7fb189ffb640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb1780a2c10 0x7fb1780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.908+0000 7fb183fff640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb1780a2c10 0x7fb1780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.908+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb17c0cf300 con 0x7fb18c100230 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.909+0000 7fb189ffb640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] conn(0x7fb1780adde0 0x7fb1780b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.909+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 --> [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb17c001d30 con 0x7fb1780adde0 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.909+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb17c0cf7e0 con 0x7fb18c100230 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.909+0000 7fb190d4b640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] conn(0x7fb1780adde0 0x7fb1780b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.909+0000 7fb183fff640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb1780a2c10 0x7fb1780a50d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb18c1043e0 tx=0x7fb17400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.909+0000 7fb190d4b640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] conn(0x7fb1780adde0 0x7fb1780b0280 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.910+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== osd.9 v2:172.21.15.64:6826/1073086660 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb17c001d30 con 0x7fb1780adde0 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 --> [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb18c062ab0 con 0x7fb1780adde0 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb189ffb640 1 -- 172.21.15.64:0/366099843 <== osd.9 v2:172.21.15.64:6826/1073086660 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb18c062ab0 con 0x7fb1780adde0 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] conn(0x7fb1780adde0 msgr2=0x7fb1780b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] conn(0x7fb1780adde0 0x7fb1780b0280 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb1780a2c10 msgr2=0x7fb1780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb1780a2c10 0x7fb1780a50d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb18c1043e0 tx=0x7fb17400a2d0 comp rx=0 tx=0).stop 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c100230 msgr2=0x7fb18c1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 --2- 172.21.15.64:0/366099843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb18c100230 0x7fb18c1980f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d770 tx=0x7fb17c00dc30 comp rx=0 tx=0).stop 2024-08-19T01:32:09.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb190d4b640 1 -- 172.21.15.64:0/366099843 reap_dead start 2024-08-19T01:32:09.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 shutdown_connections 2024-08-19T01:32:09.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 >> 172.21.15.64:0/366099843 conn(0x7fb18c0fc2f0 msgr2=0x7fb18c105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:09.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 shutdown_connections 2024-08-19T01:32:09.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:09.924+0000 7fb1927ae640 1 -- 172.21.15.64:0/366099843 wait complete. 2024-08-19T01:32:10.666 INFO:teuthology.orchestra.run.smithi064.stdout:55834574921 2024-08-19T01:32:10.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-19T01:32:11.311 INFO:teuthology.orchestra.run.smithi064.stdout:206158430254 2024-08-19T01:32:11.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-19T01:32:11.888 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:11 smithi064 ceph-mon[18515]: pgmap v309: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-19T01:32:11.914 INFO:teuthology.orchestra.run.smithi064.stdout:154618822712 2024-08-19T01:32:11.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-19T01:32:11.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:11 smithi123 ceph-mon[21829]: pgmap v309: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-19T01:32:11.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:11 smithi161 ceph-mon[21757]: pgmap v309: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-19T01:32:12.359 INFO:teuthology.orchestra.run.smithi064.stdout:330712481820 2024-08-19T01:32:12.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-08-19T01:32:12.758 INFO:teuthology.orchestra.run.smithi064.stdout:274877906982 2024-08-19T01:32:12.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-08-19T01:32:13.208 INFO:teuthology.orchestra.run.smithi064.stdout:184683593780 2024-08-19T01:32:13.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-19T01:32:13.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:13 smithi064 ceph-mon[18515]: pgmap v310: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:13.724 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:13.728 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:13.736 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:13.745 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:13.787 INFO:teuthology.orchestra.run.smithi064.stdout:77309411397 2024-08-19T01:32:13.788 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-19T01:32:13.920 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:13 smithi123 ceph-mon[21829]: pgmap v310: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:13.955 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:13 smithi161 ceph-mon[21757]: pgmap v310: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:14.075 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:14.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 -- 172.21.15.64:0/3802281077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 msgr2=0x7f99d81026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 --2- 172.21.15.64:0/3802281077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d81026e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009900 tx=0x7f99c802a480 comp rx=0 tx=0).stop 2024-08-19T01:32:14.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 -- 172.21.15.64:0/3802281077 shutdown_connections 2024-08-19T01:32:14.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 --2- 172.21.15.64:0/3802281077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99d8102c20 0x7f99d810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 --2- 172.21.15.64:0/3802281077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d81026e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 --2- 172.21.15.64:0/3802281077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99d8100b50 0x7f99d8100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 -- 172.21.15.64:0/3802281077 >> 172.21.15.64:0/3802281077 conn(0x7f99d80fc500 msgr2=0x7f99d80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 -- 172.21.15.64:0/3802281077 shutdown_connections 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.615+0000 7f99def62640 1 -- 172.21.15.64:0/3802281077 wait complete. 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 Processor -- start 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 -- start start 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99d8100b50 0x7f99d8198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99d8102c20 0x7f99d819ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8110cb0 con 0x7f99d81022a0 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8110b30 con 0x7f99d8102c20 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99def62640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8110e30 con 0x7f99d8100b50 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d8198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40576/0 (socket says 172.21.15.64:40576) 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99cffff640 1 -- 172.21.15.64:0/2042266398 learned_addr learned my addr 172.21.15.64:0/2042266398 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:14.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99dd4ff640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99d8102c20 0x7f99d819ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99dccfe640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99d8100b50 0x7f99d8198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.616+0000 7f99cffff640 1 -- 172.21.15.64:0/2042266398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99d8100b50 msgr2=0x7f99d8198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cffff640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99d8100b50 0x7f99d8198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cffff640 1 -- 172.21.15.64:0/2042266398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99d8102c20 msgr2=0x7f99d819ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cffff640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99d8102c20 0x7f99d819ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cffff640 1 -- 172.21.15.64:0/2042266398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d819d0f0 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cffff640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d8198680 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f99c8006010 tx=0x7f99c8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cdffb640 1 -- 172.21.15.64:0/2042266398 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c803f070 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cdffb640 1 -- 172.21.15.64:0/2042266398 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c8031c30 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d819d380 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99cdffb640 1 -- 172.21.15.64:0/2042266398 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8035030 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.617+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d81a4cc0 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.618+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d8101ee0 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.619+0000 7f99cdffb640 1 -- 172.21.15.64:0/2042266398 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f99c8042070 con 0x7f99d81022a0 2024-08-19T01:32:14.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.620+0000 7f99cdffb640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f99c00a2c10 0x7f99c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.620+0000 7f99dccfe640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f99c00a2c10 0x7f99c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.620+0000 7f99cdffb640 1 -- 172.21.15.64:0/2042266398 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f99c80efe40 con 0x7f99d81022a0 2024-08-19T01:32:14.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.620+0000 7f99dccfe640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f99c00a2c10 0x7f99c00a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f99d400ba40 tx=0x7f99d4005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:14.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.624+0000 7f99cdffb640 1 -- 172.21.15.64:0/2042266398 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f99c8031de0 con 0x7f99d81022a0 2024-08-19T01:32:14.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.804+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f99d8109dd0 con 0x7f99d81022a0 2024-08-19T01:32:14.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.804+0000 7f99cdffb640 1 -- 172.21.15.64:0/2042266398 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f99c80dae20 con 0x7f99d81022a0 2024-08-19T01:32:14.806 INFO:teuthology.orchestra.run.smithi064.stdout:103079215169 2024-08-19T01:32:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f99c00a2c10 msgr2=0x7f99c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f99c00a2c10 0x7f99c00a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f99d400ba40 tx=0x7f99d4005cc0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 msgr2=0x7f99d8198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d8198680 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f99c8006010 tx=0x7f99c8003c10 comp rx=0 tx=0).stop 2024-08-19T01:32:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 shutdown_connections 2024-08-19T01:32:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f99c00a2c10 0x7f99c00a50d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99d8102c20 0x7f99d819ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99d81022a0 0x7f99d8198680 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 --2- 172.21.15.64:0/2042266398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99d8100b50 0x7f99d8198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 >> 172.21.15.64:0/2042266398 conn(0x7f99d80fc500 msgr2=0x7f99d8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:14.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 shutdown_connections 2024-08-19T01:32:14.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.806+0000 7f99def62640 1 -- 172.21.15.64:0/2042266398 wait complete. 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 -- 172.21.15.64:0/1744877806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 msgr2=0x7effd41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1744877806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd41009e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7effd0009900 tx=0x7effd002a410 comp rx=0 tx=0).stop 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 -- 172.21.15.64:0/1744877806 shutdown_connections 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1744877806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effd410e1b0 0x7effd41105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1744877806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effd4100fb0 0x7effd410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1744877806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd41009e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 -- 172.21.15.64:0/1744877806 >> 172.21.15.64:0/1744877806 conn(0x7effd40fc4d0 msgr2=0x7effd40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 -- 172.21.15.64:0/1744877806 shutdown_connections 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.964+0000 7effdc1eb640 1 -- 172.21.15.64:0/1744877806 wait complete. 2024-08-19T01:32:14.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 Processor -- start 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 -- start start 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effd4100fb0 0x7effd419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effd410e1b0 0x7effd419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7effd4113ed0 con 0x7effd4100600 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7effd4113d50 con 0x7effd410e1b0 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effdc1eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7effd4114050 con 0x7effd4100fb0 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effd9f87640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effd9f87640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd419a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40602/0 (socket says 172.21.15.64:40602) 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.965+0000 7effd9f87640 1 -- 172.21.15.64:0/1559342083 learned_addr learned my addr 172.21.15.64:0/1559342083 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effd9f87640 1 -- 172.21.15.64:0/1559342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effd4100fb0 msgr2=0x7effd419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effd9786640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effd4100fb0 0x7effd419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effd9f87640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effd4100fb0 0x7effd419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effda788640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effd410e1b0 0x7effd419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effd9f87640 1 -- 172.21.15.64:0/1559342083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effd410e1b0 msgr2=0x7effd419ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effd9f87640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effd410e1b0 0x7effd419ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effd9f87640 1 -- 172.21.15.64:0/1559342083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd419f3c0 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effd9f87640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd419a410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7effd0006010 tx=0x7effd00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effcaffd640 1 -- 172.21.15.64:0/1559342083 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd003f030 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd419f650 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effcaffd640 1 -- 172.21.15.64:0/1559342083 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effd0003ce0 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effcaffd640 1 -- 172.21.15.64:0/1559342083 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0035070 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.966+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd41a6f90 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.967+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd41009e0 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.969+0000 7effcaffd640 1 -- 172.21.15.64:0/1559342083 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7effd0042030 con 0x7effd4100600 2024-08-19T01:32:14.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.970+0000 7effcaffd640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7effc00a2c10 0x7effc00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:14.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.970+0000 7effd9786640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7effc00a2c10 0x7effc00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:14.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.970+0000 7effcaffd640 1 -- 172.21.15.64:0/1559342083 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7effd00efd80 con 0x7effd4100600 2024-08-19T01:32:14.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.970+0000 7effd9786640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7effc00a2c10 0x7effc00a50d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7effc4005fd0 tx=0x7effc4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:14.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:14.973+0000 7effcaffd640 1 -- 172.21.15.64:0/1559342083 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7effd00bc8e0 con 0x7effd4100600 2024-08-19T01:32:15.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.151+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7effd4109370 con 0x7effd4100600 2024-08-19T01:32:15.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.151+0000 7effcaffd640 1 -- 172.21.15.64:0/1559342083 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7effd00bc1e0 con 0x7effd4100600 2024-08-19T01:32:15.153 INFO:teuthology.orchestra.run.smithi064.stdout:296352743458 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7effc00a2c10 msgr2=0x7effc00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7effc00a2c10 0x7effc00a50d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7effc4005fd0 tx=0x7effc4005850 comp rx=0 tx=0).stop 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 msgr2=0x7effd419a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd419a410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7effd0006010 tx=0x7effd00042f0 comp rx=0 tx=0).stop 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 shutdown_connections 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7effc00a2c10 0x7effc00a50d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effd410e1b0 0x7effd419ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effd4100fb0 0x7effd419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 --2- 172.21.15.64:0/1559342083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd4100600 0x7effd419a410 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.153+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 >> 172.21.15.64:0/1559342083 conn(0x7effd40fc4d0 msgr2=0x7effd40fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:15.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.154+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 shutdown_connections 2024-08-19T01:32:15.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:15.154+0000 7effdc1eb640 1 -- 172.21.15.64:0/1559342083 wait complete. 2024-08-19T01:32:15.859 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:15 smithi064 ceph-mon[18515]: pgmap v311: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-08-19T01:32:15.859 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:15 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2042266398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T01:32:15.859 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:15 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1559342083' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T01:32:15.890 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743455 got 296352743458 for osd.10 2024-08-19T01:32:15.890 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:15.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:15 smithi123 ceph-mon[21829]: pgmap v311: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-08-19T01:32:15.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:15 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2042266398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T01:32:15.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:15 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1559342083' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T01:32:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:15 smithi161 ceph-mon[21757]: pgmap v311: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-08-19T01:32:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:15 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2042266398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T01:32:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:15 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1559342083' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T01:32:16.577 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215167 got 103079215169 for osd.3 2024-08-19T01:32:16.577 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.832+0000 7f0845440640 1 -- 172.21.15.64:0/1285372461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 msgr2=0x7f0840107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.832+0000 7f0845440640 1 --2- 172.21.15.64:0/1285372461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f0840107c90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0830009cd0 tx=0x7f083002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:32:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.832+0000 7f0845440640 1 -- 172.21.15.64:0/1285372461 shutdown_connections 2024-08-19T01:32:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.832+0000 7f0845440640 1 --2- 172.21.15.64:0/1285372461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0840102250 0x7f084010e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.832+0000 7f0845440640 1 --2- 172.21.15.64:0/1285372461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08401018b0 0x7f0840101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.832+0000 7f0845440640 1 --2- 172.21.15.64:0/1285372461 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f0840107c90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.832+0000 7f0845440640 1 -- 172.21.15.64:0/1285372461 >> 172.21.15.64:0/1285372461 conn(0x7f08400fd5f0 msgr2=0x7f08400ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.833+0000 7f0845440640 1 -- 172.21.15.64:0/1285372461 shutdown_connections 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.833+0000 7f0845440640 1 -- 172.21.15.64:0/1285372461 wait complete. 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.833+0000 7f0845440640 1 Processor -- start 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.833+0000 7f0845440640 1 -- start start 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.833+0000 7f0845440640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08401018b0 0x7f0840198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0845440640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0840102250 0x7f0840198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0845440640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f084019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0845440640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0840111cd0 con 0x7f08401078b0 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0845440640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0840111b50 con 0x7f08401018b0 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0845440640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0840111e50 con 0x7f0840102250 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f084019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08401018b0 0x7f0840198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:16.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f084019cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40612/0 (socket says 172.21.15.64:40612) 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 -- 172.21.15.64:0/1923824 learned_addr learned my addr 172.21.15.64:0/1923824 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083e7fc640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0840102250 0x7f0840198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 -- 172.21.15.64:0/1923824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0840102250 msgr2=0x7f0840198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0840102250 0x7f0840198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 -- 172.21.15.64:0/1923824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08401018b0 msgr2=0x7f0840198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08401018b0 0x7f0840198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 -- 172.21.15.64:0/1923824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084019d1c0 con 0x7f08401078b0 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083effd640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08401018b0 0x7f0840198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f083f7fe640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f084019cae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0838009fd0 tx=0x7f083800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0837fff640 1 -- 172.21.15.64:0/1923824 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838011830 con 0x7f08401078b0 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0837fff640 1 -- 172.21.15.64:0/1923824 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0838016070 con 0x7f08401078b0 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084019d4b0 con 0x7f08401078b0 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0837fff640 1 -- 172.21.15.64:0/1923824 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083800f2e0 con 0x7f08401078b0 2024-08-19T01:32:16.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.834+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08401a4d90 con 0x7f08401078b0 2024-08-19T01:32:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.835+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0840101d10 con 0x7f08401078b0 2024-08-19T01:32:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.837+0000 7f0837fff640 1 -- 172.21.15.64:0/1923824 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f08380119d0 con 0x7f08401078b0 2024-08-19T01:32:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.837+0000 7f0837fff640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f082c0a2c10 0x7f082c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.837+0000 7f083effd640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f082c0a2c10 0x7f082c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.837+0000 7f0837fff640 1 -- 172.21.15.64:0/1923824 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f08380ce990 con 0x7f08401078b0 2024-08-19T01:32:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.838+0000 7f083effd640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f082c0a2c10 0x7f082c0a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0830006010 tx=0x7f0830005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:16.841+0000 7f0837fff640 1 -- 172.21.15.64:0/1923824 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f083809b540 con 0x7f08401078b0 2024-08-19T01:32:17.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.019+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0840109890 con 0x7f08401078b0 2024-08-19T01:32:17.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.019+0000 7f0837fff640 1 -- 172.21.15.64:0/1923824 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f083809ae40 con 0x7f08401078b0 2024-08-19T01:32:17.021 INFO:teuthology.orchestra.run.smithi064.stdout:133143986239 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f082c0a2c10 msgr2=0x7f082c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f082c0a2c10 0x7f082c0a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0830006010 tx=0x7f0830005ef0 comp rx=0 tx=0).stop 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 msgr2=0x7f084019cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f084019cae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0838009fd0 tx=0x7f083800b8e0 comp rx=0 tx=0).stop 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 shutdown_connections 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f082c0a2c10 0x7f082c0a50d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08401078b0 0x7f084019cae0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0840102250 0x7f0840198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 --2- 172.21.15.64:0/1923824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08401018b0 0x7f0840198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 >> 172.21.15.64:0/1923824 conn(0x7f08400fd5f0 msgr2=0x7f08400fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 shutdown_connections 2024-08-19T01:32:17.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:17.021+0000 7f0845440640 1 -- 172.21.15.64:0/1923824 wait complete. 2024-08-19T01:32:17.706 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:17 smithi064 ceph-mon[18515]: pgmap v312: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:17.706 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:17 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1923824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T01:32:17.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986236 got 133143986239 for osd.4 2024-08-19T01:32:17.736 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:17 smithi123 ceph-mon[21829]: pgmap v312: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:17.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:17 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1923824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T01:32:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:17 smithi161 ceph-mon[21757]: pgmap v312: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:17 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1923824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T01:32:17.958 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:17.959 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 -- 172.21.15.64:0/1401088234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1501022a0 msgr2=0x7fa1501026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 --2- 172.21.15.64:0/1401088234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1501022a0 0x7fa1501026e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa140009900 tx=0x7fa14002a480 comp rx=0 tx=0).stop 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 -- 172.21.15.64:0/1401088234 shutdown_connections 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 --2- 172.21.15.64:0/1401088234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa150102c20 0x7fa15010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 --2- 172.21.15.64:0/1401088234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1501022a0 0x7fa1501026e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 --2- 172.21.15.64:0/1401088234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa150100b50 0x7fa150100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 -- 172.21.15.64:0/1401088234 >> 172.21.15.64:0/1401088234 conn(0x7fa1500fc500 msgr2=0x7fa1500fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 -- 172.21.15.64:0/1401088234 shutdown_connections 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 -- 172.21.15.64:0/1401088234 wait complete. 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 Processor -- start 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.166+0000 7fa157afa640 1 -- start start 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa157afa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa150100b50 0x7fa150196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa157afa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501022a0 0x7fa150197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa157afa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa150102c20 0x7fa150197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa157afa640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa150110cf0 con 0x7fa150102c20 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa157afa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa150110b70 con 0x7fa150100b50 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa157afa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa150110e70 con 0x7fa1501022a0 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa150102c20 0x7fa150197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa150102c20 0x7fa150197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40626/0 (socket says 172.21.15.64:40626) 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa155095640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501022a0 0x7fa150197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 -- 172.21.15.64:0/1400955886 learned_addr learned my addr 172.21.15.64:0/1400955886 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa155896640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa150100b50 0x7fa150196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 -- 172.21.15.64:0/1400955886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501022a0 msgr2=0x7fa150197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501022a0 0x7fa150197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 -- 172.21.15.64:0/1400955886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa150100b50 msgr2=0x7fa150196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa150100b50 0x7fa150196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 -- 172.21.15.64:0/1400955886 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1501a6740 con 0x7fa150102c20 2024-08-19T01:32:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.167+0000 7fa156097640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa150102c20 0x7fa150197950 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa14800cad0 tx=0x7fa14800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.168+0000 7fa146ffd640 1 -- 172.21.15.64:0/1400955886 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14800ad60 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.168+0000 7fa146ffd640 1 -- 172.21.15.64:0/1400955886 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa148005020 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.168+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1501a6a30 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.168+0000 7fa146ffd640 1 -- 172.21.15.64:0/1400955886 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14800f6b0 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.168+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1501a6fc0 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.169+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa150100f30 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.170+0000 7fa146ffd640 1 -- 172.21.15.64:0/1400955886 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa148005560 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.170+0000 7fa146ffd640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa13c0a2c10 0x7fa13c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.170+0000 7fa155896640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa13c0a2c10 0x7fa13c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.170+0000 7fa146ffd640 1 -- 172.21.15.64:0/1400955886 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa1480cf480 con 0x7fa150102c20 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.171+0000 7fa155896640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa13c0a2c10 0x7fa13c0a50d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa14c009990 tx=0x7fa14c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.174+0000 7fa146ffd640 1 -- 172.21.15.64:0/1400955886 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa148014030 con 0x7fa150102c20 2024-08-19T01:32:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.359+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fa150108dc0 con 0x7fa150102c20 2024-08-19T01:32:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.359+0000 7fa146ffd640 1 -- 172.21.15.64:0/1400955886 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa14809bf20 con 0x7fa150102c20 2024-08-19T01:32:18.361 INFO:teuthology.orchestra.run.smithi064.stdout:240518168618 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa13c0a2c10 msgr2=0x7fa13c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa13c0a2c10 0x7fa13c0a50d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa14c009990 tx=0x7fa14c009110 comp rx=0 tx=0).stop 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa150102c20 msgr2=0x7fa150197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa150102c20 0x7fa150197950 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa14800cad0 tx=0x7fa14800ced0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 shutdown_connections 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa13c0a2c10 0x7fa13c0a50d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa150102c20 0x7fa150197950 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501022a0 0x7fa150197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 --2- 172.21.15.64:0/1400955886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa150100b50 0x7fa150196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 >> 172.21.15.64:0/1400955886 conn(0x7fa1500fc500 msgr2=0x7fa1500fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.361+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 shutdown_connections 2024-08-19T01:32:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:18.362+0000 7fa157afa640 1 -- 172.21.15.64:0/1400955886 wait complete. 2024-08-19T01:32:18.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:18 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1400955886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T01:32:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:18 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1400955886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T01:32:18.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:18 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1400955886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T01:32:19.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168616 got 240518168618 for osd.8 2024-08-19T01:32:19.149 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:19.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1790077316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4102ac0 msgr2=0x7f2ef4102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1790077316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4102ac0 0x7f2ef4102ea0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0009930 tx=0x7f2ef002a420 comp rx=0 tx=0).stop 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1790077316 shutdown_connections 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1790077316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef4107fc0 0x7f2ef410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1790077316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef4103470 0x7f2ef41078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1790077316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4102ac0 0x7f2ef4102ea0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1790077316 >> 172.21.15.64:0/1790077316 conn(0x7f2ef40fc500 msgr2=0x7f2ef40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1790077316 shutdown_connections 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.328+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1790077316 wait complete. 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 Processor -- start 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 -- start start 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef4102ac0 0x7f2ef4100720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef4103470 0x7f2ef4100c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4107fc0 0x7f2ef41011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:19.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef4111dc0 con 0x7f2ef4107fc0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef4111c40 con 0x7f2ef4103470 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efc4e1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef4111f40 con 0x7f2ef4102ac0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efaa7e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4107fc0 0x7f2ef41011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efaa7e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4107fc0 0x7f2ef41011a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53256/0 (socket says 172.21.15.64:53256) 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.329+0000 7f2efaa7e640 1 -- 172.21.15.64:0/1811154084 learned_addr learned my addr 172.21.15.64:0/1811154084 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2ef9a7c640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef4103470 0x7f2ef4100c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efa27d640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef4102ac0 0x7f2ef4100720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efaa7e640 1 -- 172.21.15.64:0/1811154084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef4102ac0 msgr2=0x7f2ef4100720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efaa7e640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef4102ac0 0x7f2ef4100720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efaa7e640 1 -- 172.21.15.64:0/1811154084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef4103470 msgr2=0x7f2ef4100c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efaa7e640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef4103470 0x7f2ef4100c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efaa7e640 1 -- 172.21.15.64:0/1811154084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef41a67e0 con 0x7f2ef4107fc0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efaa7e640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4107fc0 0x7f2ef41011a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009fd0 tx=0x7f2eec00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2eeb7fe640 1 -- 172.21.15.64:0/1811154084 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec017070 con 0x7f2ef4107fc0 2024-08-19T01:32:19.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2eeb7fe640 1 -- 172.21.15.64:0/1811154084 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eec003e50 con 0x7f2ef4107fc0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef41a6ad0 con 0x7f2ef4107fc0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2eeb7fe640 1 -- 172.21.15.64:0/1811154084 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0106c0 con 0x7f2ef4107fc0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.330+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef41a7010 con 0x7f2ef4107fc0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.331+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ef406a520 con 0x7f2ef4107fc0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.332+0000 7f2eeb7fe640 1 -- 172.21.15.64:0/1811154084 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2eec003970 con 0x7f2ef4107fc0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.333+0000 7f2eeb7fe640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ee00a2c90 0x7f2ee00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.333+0000 7f2efa27d640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ee00a2c90 0x7f2ee00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.333+0000 7f2eeb7fe640 1 -- 172.21.15.64:0/1811154084 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2eec0cf280 con 0x7f2ef4107fc0 2024-08-19T01:32:19.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.334+0000 7f2efa27d640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ee00a2c90 0x7f2ee00a5150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0000c00 tx=0x7f2ef0005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:19.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.339+0000 7f2eeb7fe640 1 -- 172.21.15.64:0/1811154084 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f2eec09bd20 con 0x7f2ef4107fc0 2024-08-19T01:32:19.344 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.524+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2ef4109fd0 con 0x7f2ef4107fc0 2024-08-19T01:32:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.524+0000 7f2eeb7fe640 1 -- 172.21.15.64:0/1811154084 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2eec09b620 con 0x7f2ef4107fc0 2024-08-19T01:32:19.526 INFO:teuthology.orchestra.run.smithi064.stdout:206158430256 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ee00a2c90 msgr2=0x7f2ee00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ee00a2c90 0x7f2ee00a5150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0000c00 tx=0x7f2ef0005a20 comp rx=0 tx=0).stop 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4107fc0 msgr2=0x7f2ef41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4107fc0 0x7f2ef41011a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009fd0 tx=0x7f2eec00b260 comp rx=0 tx=0).stop 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 shutdown_connections 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2ee00a2c90 0x7f2ee00a5150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ef4107fc0 0x7f2ef41011a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef4103470 0x7f2ef4100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 --2- 172.21.15.64:0/1811154084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef4102ac0 0x7f2ef4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.526+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 >> 172.21.15.64:0/1811154084 conn(0x7f2ef40fc500 msgr2=0x7f2ef4105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.527+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 shutdown_connections 2024-08-19T01:32:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:19.527+0000 7f2efc4e1640 1 -- 172.21.15.64:0/1811154084 wait complete. 2024-08-19T01:32:19.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:19 smithi064 ceph-mon[18515]: pgmap v313: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:19.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:19 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1811154084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T01:32:19.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:19 smithi123 ceph-mon[21829]: pgmap v313: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:19.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:19 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1811154084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T01:32:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:19 smithi161 ceph-mon[21757]: pgmap v313: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:19 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1811154084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T01:32:20.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 -- 172.21.15.64:0/3399934687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8100b50 msgr2=0x7f16d8100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:20.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 --2- 172.21.15.64:0/3399934687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8100b50 0x7f16d8100f30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f16d4009900 tx=0x7f16d402a480 comp rx=0 tx=0).stop 2024-08-19T01:32:20.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 -- 172.21.15.64:0/3399934687 shutdown_connections 2024-08-19T01:32:20.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 --2- 172.21.15.64:0/3399934687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16d8102c20 0x7f16d810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 --2- 172.21.15.64:0/3399934687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d81022a0 0x7f16d81026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 --2- 172.21.15.64:0/3399934687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8100b50 0x7f16d8100f30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 -- 172.21.15.64:0/3399934687 >> 172.21.15.64:0/3399934687 conn(0x7f16d80fc500 msgr2=0x7f16d80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 -- 172.21.15.64:0/3399934687 shutdown_connections 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.157+0000 7f16e023e640 1 -- 172.21.15.64:0/3399934687 wait complete. 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 Processor -- start 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 -- start start 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16d8100b50 0x7f16d80fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d81022a0 0x7f16d80fcef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8102c20 0x7f16d80fd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8110d50 con 0x7f16d8102c20 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8110bd0 con 0x7f16d81022a0 2024-08-19T01:32:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16e023e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8110ed0 con 0x7f16d8100b50 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16de7db640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8102c20 0x7f16d80fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16de7db640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8102c20 0x7f16d80fd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53274/0 (socket says 172.21.15.64:53274) 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.158+0000 7f16de7db640 1 -- 172.21.15.64:0/1123175386 learned_addr learned my addr 172.21.15.64:0/1123175386 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16dd7d9640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d81022a0 0x7f16d80fcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16ddfda640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16d8100b50 0x7f16d80fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16de7db640 1 -- 172.21.15.64:0/1123175386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16d8100b50 msgr2=0x7f16d80fc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16de7db640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16d8100b50 0x7f16d80fc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16de7db640 1 -- 172.21.15.64:0/1123175386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d81022a0 msgr2=0x7f16d80fcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16de7db640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d81022a0 0x7f16d80fcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16de7db640 1 -- 172.21.15.64:0/1123175386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d81a6860 con 0x7f16d8102c20 2024-08-19T01:32:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16de7db640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8102c20 0x7f16d80fd430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f16d000b970 tx=0x7f16d000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:20.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16ceffd640 1 -- 172.21.15.64:0/1123175386 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0019070 con 0x7f16d8102c20 2024-08-19T01:32:20.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16ceffd640 1 -- 172.21.15.64:0/1123175386 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16d0014440 con 0x7f16d8102c20 2024-08-19T01:32:20.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d81a6b50 con 0x7f16d8102c20 2024-08-19T01:32:20.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16ceffd640 1 -- 172.21.15.64:0/1123175386 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d00075d0 con 0x7f16d8102c20 2024-08-19T01:32:20.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.159+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d81a7090 con 0x7f16d8102c20 2024-08-19T01:32:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.160+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d8100fb0 con 0x7f16d8102c20 2024-08-19T01:32:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.162+0000 7f16ceffd640 1 -- 172.21.15.64:0/1123175386 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f16d001c070 con 0x7f16d8102c20 2024-08-19T01:32:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.162+0000 7f16ceffd640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f16c40a2c10 0x7f16c40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.162+0000 7f16ddfda640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f16c40a2c10 0x7f16c40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.162+0000 7f16ceffd640 1 -- 172.21.15.64:0/1123175386 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f16d00cea20 con 0x7f16d8102c20 2024-08-19T01:32:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.163+0000 7f16ddfda640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f16c40a2c10 0x7f16c40a50d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005ab0 tx=0x7f16d4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:20.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.167+0000 7f16ceffd640 1 -- 172.21.15.64:0/1123175386 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f16d009b5d0 con 0x7f16d8102c20 2024-08-19T01:32:20.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.344+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f16d8109dd0 con 0x7f16d8102c20 2024-08-19T01:32:20.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.345+0000 7f16ceffd640 1 -- 172.21.15.64:0/1123175386 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f16d009aed0 con 0x7f16d8102c20 2024-08-19T01:32:20.346 INFO:teuthology.orchestra.run.smithi064.stdout:34359738447 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.346+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f16c40a2c10 msgr2=0x7f16c40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.346+0000 7f16e023e640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f16c40a2c10 0x7f16c40a50d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005ab0 tx=0x7f16d4005a20 comp rx=0 tx=0).stop 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.346+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8102c20 msgr2=0x7f16d80fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.346+0000 7f16e023e640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8102c20 0x7f16d80fd430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f16d000b970 tx=0x7f16d000bd70 comp rx=0 tx=0).stop 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 shutdown_connections 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f16c40a2c10 0x7f16c40a50d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16d8102c20 0x7f16d80fd430 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d81022a0 0x7f16d80fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 --2- 172.21.15.64:0/1123175386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16d8100b50 0x7f16d80fc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:20.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 >> 172.21.15.64:0/1123175386 conn(0x7f16d80fc500 msgr2=0x7f16d8069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:20.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 shutdown_connections 2024-08-19T01:32:20.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:20.347+0000 7f16e023e640 1 -- 172.21.15.64:0/1123175386 wait complete. 2024-08-19T01:32:20.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:20 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1123175386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T01:32:20.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:20 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1123175386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T01:32:20.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:20 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1123175386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T01:32:21.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 -- 172.21.15.64:0/853979993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848106690 msgr2=0x7ff848106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 --2- 172.21.15.64:0/853979993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848106690 0x7ff848106a70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff838009930 tx=0x7ff83802a420 comp rx=0 tx=0).stop 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 -- 172.21.15.64:0/853979993 shutdown_connections 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 --2- 172.21.15.64:0/853979993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff848100fc0 0x7ff84810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 --2- 172.21.15.64:0/853979993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff848100600 0x7ff848100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 --2- 172.21.15.64:0/853979993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848106690 0x7ff848106a70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 -- 172.21.15.64:0/853979993 >> 172.21.15.64:0/853979993 conn(0x7ff8480fc360 msgr2=0x7ff8480fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 -- 172.21.15.64:0/853979993 shutdown_connections 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.057+0000 7ff84d79e640 1 -- 172.21.15.64:0/853979993 wait complete. 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 Processor -- start 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 -- start start 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff848100600 0x7ff848198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848100fc0 0x7ff848199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff848106690 0x7ff8481a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481109f0 con 0x7ff848100fc0 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff848110870 con 0x7ff848106690 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff84d79e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff848110b70 con 0x7ff848100600 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff8477fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848100fc0 0x7ff848199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff8477fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848100fc0 0x7ff848199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53294/0 (socket says 172.21.15.64:53294) 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.058+0000 7ff8477fe640 1 -- 172.21.15.64:0/2631194220 learned_addr learned my addr 172.21.15.64:0/2631194220 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff847fff640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff848100600 0x7ff848198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8477fe640 1 -- 172.21.15.64:0/2631194220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff848100600 msgr2=0x7ff848198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8477fe640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff848100600 0x7ff848198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8477fe640 1 -- 172.21.15.64:0/2631194220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff848106690 msgr2=0x7ff8481a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff84cf9d640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff848106690 0x7ff8481a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8477fe640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff848106690 0x7ff8481a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8477fe640 1 -- 172.21.15.64:0/2631194220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8481a43a0 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff84cf9d640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff848106690 0x7ff8481a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8477fe640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848100fc0 0x7ff848199320 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00d770 tx=0x7ff83c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8457fa640 1 -- 172.21.15.64:0/2631194220 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff83c017070 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8457fa640 1 -- 172.21.15.64:0/2631194220 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff83c005500 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8481a4570 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff8457fa640 1 -- 172.21.15.64:0/2631194220 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff83c004e00 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.059+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8481a4a30 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.060+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff848100a80 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.061+0000 7ff8457fa640 1 -- 172.21.15.64:0/2631194220 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff83c004fa0 con 0x7ff848100fc0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.062+0000 7ff8457fa640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8300a2c10 0x7ff8300a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.062+0000 7ff847fff640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8300a2c10 0x7ff8300a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.062+0000 7ff8457fa640 1 -- 172.21.15.64:0/2631194220 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff83c0cf060 con 0x7ff848100fc0 2024-08-19T01:32:21.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.062+0000 7ff847fff640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8300a2c10 0x7ff8300a50d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff838000c00 tx=0x7ff838005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:21.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.066+0000 7ff8457fa640 1 -- 172.21.15.64:0/2631194220 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff83c09bb00 con 0x7ff848100fc0 2024-08-19T01:32:21.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.243+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff848108b70 con 0x7ff848100fc0 2024-08-19T01:32:21.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.244+0000 7ff8457fa640 1 -- 172.21.15.64:0/2631194220 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff83c09b400 con 0x7ff848100fc0 2024-08-19T01:32:21.245 INFO:teuthology.orchestra.run.smithi064.stdout:55834574923 2024-08-19T01:32:21.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8300a2c10 msgr2=0x7ff8300a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8300a2c10 0x7ff8300a50d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff838000c00 tx=0x7ff838005a20 comp rx=0 tx=0).stop 2024-08-19T01:32:21.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848100fc0 msgr2=0x7ff848199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848100fc0 0x7ff848199320 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00d770 tx=0x7ff83c00dc30 comp rx=0 tx=0).stop 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 shutdown_connections 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8300a2c10 0x7ff8300a50d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff848106690 0x7ff8481a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff848100fc0 0x7ff848199320 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 --2- 172.21.15.64:0/2631194220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff848100600 0x7ff848198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 >> 172.21.15.64:0/2631194220 conn(0x7ff8480fc360 msgr2=0x7ff8480fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 shutdown_connections 2024-08-19T01:32:21.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.246+0000 7ff84d79e640 1 -- 172.21.15.64:0/2631194220 wait complete. 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 -- 172.21.15.64:0/3931266245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff900101150 msgr2=0x7ff90010d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 --2- 172.21.15.64:0/3931266245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff900101150 0x7ff90010d680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009cd0 tx=0x7ff8f802a8e0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 -- 172.21.15.64:0/3931266245 shutdown_connections 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 --2- 172.21.15.64:0/3931266245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff900101150 0x7ff90010d680 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 --2- 172.21.15.64:0/3931266245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9001007b0 0x7ff900100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 --2- 172.21.15.64:0/3931266245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9001067b0 0x7ff900106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 -- 172.21.15.64:0/3931266245 >> 172.21.15.64:0/3931266245 conn(0x7ff9000fc4d0 msgr2=0x7ff9000fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 -- 172.21.15.64:0/3931266245 shutdown_connections 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.796+0000 7ff905191640 1 -- 172.21.15.64:0/3931266245 wait complete. 2024-08-19T01:32:21.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 Processor -- start 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 -- start start 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9001007b0 0x7ff90010e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff900101150 0x7ff90010e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9001067b0 0x7ff90010eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff900111d30 con 0x7ff9001007b0 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff900111bb0 con 0x7ff9001067b0 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff905191640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff900111eb0 con 0x7ff900101150 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9001007b0 0x7ff90010e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9001007b0 0x7ff90010e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53302/0 (socket says 172.21.15.64:53302) 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fed9d640 1 -- 172.21.15.64:0/3756578247 learned_addr learned my addr 172.21.15.64:0/3756578247 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8ff59e640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9001067b0 0x7ff90010eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fe59c640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff900101150 0x7ff90010e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fed9d640 1 -- 172.21.15.64:0/3756578247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff900101150 msgr2=0x7ff90010e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fed9d640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff900101150 0x7ff90010e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fed9d640 1 -- 172.21.15.64:0/3756578247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9001067b0 msgr2=0x7ff90010eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.797+0000 7ff8fed9d640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9001067b0 0x7ff90010eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff8fed9d640 1 -- 172.21.15.64:0/3756578247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9001a6670 con 0x7ff9001007b0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff8ff59e640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9001067b0 0x7ff90010eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff8fe59c640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff900101150 0x7ff90010e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff8fed9d640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9001007b0 0x7ff90010e110 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000c890 tx=0x7ff8f000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff8e7fff640 1 -- 172.21.15.64:0/3756578247 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0004220 con 0x7ff9001007b0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9001a6960 con 0x7ff9001007b0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff8e7fff640 1 -- 172.21.15.64:0/3756578247 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8f00043c0 con 0x7ff9001007b0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff8e7fff640 1 -- 172.21.15.64:0/3756578247 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0004a00 con 0x7ff9001007b0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.798+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9001a6ef0 con 0x7ff9001007b0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.799+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff900100c10 con 0x7ff9001007b0 2024-08-19T01:32:21.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.800+0000 7ff8e7fff640 1 -- 172.21.15.64:0/3756578247 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff8f00079b0 con 0x7ff9001007b0 2024-08-19T01:32:21.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.801+0000 7ff8e7fff640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e80a2c10 0x7ff8e80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:21.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.801+0000 7ff8fe59c640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e80a2c10 0x7ff8e80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:21.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.801+0000 7ff8e7fff640 1 -- 172.21.15.64:0/3756578247 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff8f00cf2e0 con 0x7ff9001007b0 2024-08-19T01:32:21.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.801+0000 7ff8fe59c640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e80a2c10 0x7ff8e80a50d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4006fd0 tx=0x7ff8f4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:21.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.805+0000 7ff8e7fff640 1 -- 172.21.15.64:0/3756578247 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff8f0014030 con 0x7ff9001007b0 2024-08-19T01:32:21.869 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:21 smithi064 ceph-mon[18515]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-08-19T01:32:21.869 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:21 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2631194220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T01:32:21.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:21 smithi123 ceph-mon[21829]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-08-19T01:32:21.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:21 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2631194220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T01:32:21.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:21 smithi161 ceph-mon[21757]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-08-19T01:32:21.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:21 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2631194220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T01:32:21.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.983+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff9001a7180 con 0x7ff9001007b0 2024-08-19T01:32:21.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.983+0000 7ff8e7fff640 1 -- 172.21.15.64:0/3756578247 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff8f009bd80 con 0x7ff9001007b0 2024-08-19T01:32:21.986 INFO:teuthology.orchestra.run.smithi064.stdout:184683593782 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e80a2c10 msgr2=0x7ff8e80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e80a2c10 0x7ff8e80a50d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4006fd0 tx=0x7ff8f4008040 comp rx=0 tx=0).stop 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9001007b0 msgr2=0x7ff90010e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9001007b0 0x7ff90010e110 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000c890 tx=0x7ff8f000cd50 comp rx=0 tx=0).stop 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 shutdown_connections 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e80a2c10 0x7ff8e80a50d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9001067b0 0x7ff90010eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff900101150 0x7ff90010e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 --2- 172.21.15.64:0/3756578247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9001007b0 0x7ff90010e110 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 >> 172.21.15.64:0/3756578247 conn(0x7ff9000fc4d0 msgr2=0x7ff9000fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 shutdown_connections 2024-08-19T01:32:21.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:21.985+0000 7ff905191640 1 -- 172.21.15.64:0/3756578247 wait complete. 2024-08-19T01:32:22.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:22 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3756578247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T01:32:22.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430254 got 206158430256 for osd.7 2024-08-19T01:32:22.786 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:22.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:22 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3756578247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T01:32:22.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:22 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3756578247' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T01:32:22.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 -- 172.21.15.64:0/3767228781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20100b50 msgr2=0x7f8c20100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:22.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/3767228781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20100b50 0x7f8c20100f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009900 tx=0x7f8c1002a480 comp rx=0 tx=0).stop 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 -- 172.21.15.64:0/3767228781 shutdown_connections 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/3767228781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c20102c20 0x7f8c2010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/3767228781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c201022a0 0x7f8c201026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/3767228781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20100b50 0x7f8c20100f30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 -- 172.21.15.64:0/3767228781 >> 172.21.15.64:0/3767228781 conn(0x7f8c200fc500 msgr2=0x7f8c200fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 -- 172.21.15.64:0/3767228781 shutdown_connections 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.948+0000 7f8c25c7d640 1 -- 172.21.15.64:0/3767228781 wait complete. 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 Processor -- start 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 -- start start 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c20100b50 0x7f8c20198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c201022a0 0x7f8c201986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:22.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20102c20 0x7f8c2019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20110cb0 con 0x7f8c20102c20 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20110b30 con 0x7f8c20100b50 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c25c7d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20110e30 con 0x7f8c201022a0 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c1ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20102c20 0x7f8c2019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c1ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20102c20 0x7f8c2019ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53326/0 (socket says 172.21.15.64:53326) 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c1ffff640 1 -- 172.21.15.64:0/195090191 learned_addr learned my addr 172.21.15.64:0/195090191 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c1f7fe640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c20100b50 0x7f8c20198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.949+0000 7f8c16ffd640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c201022a0 0x7f8c201986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1ffff640 1 -- 172.21.15.64:0/195090191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c201022a0 msgr2=0x7f8c201986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1ffff640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c201022a0 0x7f8c201986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1ffff640 1 -- 172.21.15.64:0/195090191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c20100b50 msgr2=0x7f8c20198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1ffff640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c20100b50 0x7f8c20198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1ffff640 1 -- 172.21.15.64:0/195090191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2019d140 con 0x7f8c20102c20 2024-08-19T01:32:22.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c16ffd640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c201022a0 0x7f8c201986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1ffff640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20102c20 0x7f8c2019ca60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800cad0 tx=0x7f8c1800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1d7fa640 1 -- 172.21.15.64:0/195090191 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800ae20 con 0x7f8c20102c20 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2019d430 con 0x7f8c20102c20 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1d7fa640 1 -- 172.21.15.64:0/195090191 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c18005020 con 0x7f8c20102c20 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c1d7fa640 1 -- 172.21.15.64:0/195090191 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800f6b0 con 0x7f8c20102c20 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.950+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c201a4d10 con 0x7f8c20102c20 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.951+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c20100f30 con 0x7f8c20102c20 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.952+0000 7f8c1d7fa640 1 -- 172.21.15.64:0/195090191 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8c180051d0 con 0x7f8c20102c20 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.953+0000 7f8c1d7fa640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8c0c0a2c10 0x7f8c0c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.953+0000 7f8c1f7fe640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8c0c0a2c10 0x7f8c0c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:22.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.953+0000 7f8c1d7fa640 1 -- 172.21.15.64:0/195090191 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8c180cf8c0 con 0x7f8c20102c20 2024-08-19T01:32:22.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.953+0000 7f8c1f7fe640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8c0c0a2c10 0x7f8c0c0a50d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000c00 tx=0x7f8c100058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:22.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:22.957+0000 7f8c1d7fa640 1 -- 172.21.15.64:0/195090191 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f8c18014030 con 0x7f8c20102c20 2024-08-19T01:32:23.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.135+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f8c20109dd0 con 0x7f8c20102c20 2024-08-19T01:32:23.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.135+0000 7f8c1d7fa640 1 -- 172.21.15.64:0/195090191 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8c1809c360 con 0x7f8c20102c20 2024-08-19T01:32:23.138 INFO:teuthology.orchestra.run.smithi064.stdout:154618822715 2024-08-19T01:32:23.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.137+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8c0c0a2c10 msgr2=0x7f8c0c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.137+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8c0c0a2c10 0x7f8c0c0a50d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000c00 tx=0x7f8c100058d0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.137+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20102c20 msgr2=0x7f8c2019ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.137+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20102c20 0x7f8c2019ca60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800cad0 tx=0x7f8c1800cf90 comp rx=0 tx=0).stop 2024-08-19T01:32:23.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 shutdown_connections 2024-08-19T01:32:23.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8c0c0a2c10 0x7f8c0c0a50d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c20102c20 0x7f8c2019ca60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c201022a0 0x7f8c201986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 --2- 172.21.15.64:0/195090191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c20100b50 0x7f8c20198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 >> 172.21.15.64:0/195090191 conn(0x7f8c200fc500 msgr2=0x7f8c20069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:23.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 shutdown_connections 2024-08-19T01:32:23.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.138+0000 7f8c25c7d640 1 -- 172.21.15.64:0/195090191 wait complete. 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.155+0000 7f8525bff640 1 -- 172.21.15.64:0/1879054307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520102ac0 msgr2=0x7f8520102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.155+0000 7f8525bff640 1 --2- 172.21.15.64:0/1879054307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520102ac0 0x7f8520102ea0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8510009930 tx=0x7f851002a420 comp rx=0 tx=0).stop 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.155+0000 7f8525bff640 1 -- 172.21.15.64:0/1879054307 shutdown_connections 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.155+0000 7f8525bff640 1 --2- 172.21.15.64:0/1879054307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8520107fc0 0x7f852010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.155+0000 7f8525bff640 1 --2- 172.21.15.64:0/1879054307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8520103470 0x7f85201078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.155+0000 7f8525bff640 1 --2- 172.21.15.64:0/1879054307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520102ac0 0x7f8520102ea0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.156+0000 7f8525bff640 1 -- 172.21.15.64:0/1879054307 >> 172.21.15.64:0/1879054307 conn(0x7f85200fc500 msgr2=0x7f85200fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.156+0000 7f8525bff640 1 -- 172.21.15.64:0/1879054307 shutdown_connections 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.156+0000 7f8525bff640 1 -- 172.21.15.64:0/1879054307 wait complete. 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.156+0000 7f8525bff640 1 Processor -- start 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f8525bff640 1 -- start start 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f8525bff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8520102ac0 0x7f8520198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f8525bff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520103470 0x7f8520198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f8525bff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8520107fc0 0x7f852019cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f8525bff640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8520111dc0 con 0x7f8520103470 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f8525bff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8520111c40 con 0x7f8520107fc0 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f8525bff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8520111f40 con 0x7f8520102ac0 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520103470 0x7f8520198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520103470 0x7f8520198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53336/0 (socket says 172.21.15.64:53336) 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 -- 172.21.15.64:0/1039866076 learned_addr learned my addr 172.21.15.64:0/1039866076 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851ffff640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8520107fc0 0x7f852019cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851f7fe640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8520102ac0 0x7f8520198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 -- 172.21.15.64:0/1039866076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8520102ac0 msgr2=0x7f8520198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8520102ac0 0x7f8520198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 -- 172.21.15.64:0/1039866076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8520107fc0 msgr2=0x7f852019cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8520107fc0 0x7f852019cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.157+0000 7f851effd640 1 -- 172.21.15.64:0/1039866076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852019d200 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.158+0000 7f851effd640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520103470 0x7f8520198790 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f851400c970 tx=0x7f851400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.158+0000 7f851cff9640 1 -- 172.21.15.64:0/1039866076 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514014070 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.158+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852019d4f0 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.158+0000 7f851cff9640 1 -- 172.21.15.64:0/1039866076 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8514004440 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.158+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85201a4dd0 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.158+0000 7f851cff9640 1 -- 172.21.15.64:0/1039866076 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851400f450 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.159+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f852006a520 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.160+0000 7f851cff9640 1 -- 172.21.15.64:0/1039866076 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8514017070 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.161+0000 7f851cff9640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85080a2c10 0x7f85080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.161+0000 7f851cff9640 1 -- 172.21.15.64:0/1039866076 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f85140cf7e0 con 0x7f8520103470 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.161+0000 7f851f7fe640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85080a2c10 0x7f85080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.162+0000 7f851f7fe640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85080a2c10 0x7f85080a50d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8510000c00 tx=0x7f8510005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:23.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.168+0000 7f851cff9640 1 -- 172.21.15.64:0/1039866076 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f851401e020 con 0x7f8520103470 2024-08-19T01:32:23.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.347+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f8520102ea0 con 0x7f8520103470 2024-08-19T01:32:23.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.347+0000 7f851cff9640 1 -- 172.21.15.64:0/1039866076 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f8520102ea0 con 0x7f8520103470 2024-08-19T01:32:23.350 INFO:teuthology.orchestra.run.smithi064.stdout:330712481822 2024-08-19T01:32:23.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85080a2c10 msgr2=0x7f85080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85080a2c10 0x7f85080a50d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8510000c00 tx=0x7f8510005a20 comp rx=0 tx=0).stop 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520103470 msgr2=0x7f8520198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520103470 0x7f8520198790 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f851400c970 tx=0x7f851400ce30 comp rx=0 tx=0).stop 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 shutdown_connections 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f85080a2c10 0x7f85080a50d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8520107fc0 0x7f852019cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520103470 0x7f8520198790 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 --2- 172.21.15.64:0/1039866076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8520102ac0 0x7f8520198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 >> 172.21.15.64:0/1039866076 conn(0x7f85200fc500 msgr2=0x7f85201060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 shutdown_connections 2024-08-19T01:32:23.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.350+0000 7f8525bff640 1 -- 172.21.15.64:0/1039866076 wait complete. 2024-08-19T01:32:23.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.385+0000 7f639c54b640 1 -- 172.21.15.64:0/2787799476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394103470 msgr2=0x7f63941078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.385+0000 7f639c54b640 1 --2- 172.21.15.64:0/2787799476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394103470 0x7f63941078b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6384009930 tx=0x7f638402a420 comp rx=0 tx=0).stop 2024-08-19T01:32:23.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.385+0000 7f639c54b640 1 -- 172.21.15.64:0/2787799476 shutdown_connections 2024-08-19T01:32:23.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.385+0000 7f639c54b640 1 --2- 172.21.15.64:0/2787799476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6394107fc0 0x7f639410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.385+0000 7f639c54b640 1 --2- 172.21.15.64:0/2787799476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394103470 0x7f63941078b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.385+0000 7f639c54b640 1 --2- 172.21.15.64:0/2787799476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6394102ac0 0x7f6394102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.385+0000 7f639c54b640 1 -- 172.21.15.64:0/2787799476 >> 172.21.15.64:0/2787799476 conn(0x7f63940fc500 msgr2=0x7f63940fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.386+0000 7f639c54b640 1 -- 172.21.15.64:0/2787799476 shutdown_connections 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.386+0000 7f639c54b640 1 -- 172.21.15.64:0/2787799476 wait complete. 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.386+0000 7f639c54b640 1 Processor -- start 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639c54b640 1 -- start start 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639c54b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6394102ac0 0x7f6394198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639c54b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6394103470 0x7f6394198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639c54b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394107fc0 0x7f639419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639c54b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111dc0 con 0x7f6394107fc0 2024-08-19T01:32:23.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639c54b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111c40 con 0x7f6394102ac0 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639c54b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111f40 con 0x7f6394103470 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394107fc0 0x7f639419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394107fc0 0x7f639419cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53356/0 (socket says 172.21.15.64:53356) 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 -- 172.21.15.64:0/2075470483 learned_addr learned my addr 172.21.15.64:0/2075470483 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639a2e7640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6394102ac0 0x7f6394198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f6399ae6640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6394103470 0x7f6394198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 -- 172.21.15.64:0/2075470483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6394103470 msgr2=0x7f6394198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6394103470 0x7f6394198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 -- 172.21.15.64:0/2075470483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6394102ac0 msgr2=0x7f6394198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6394102ac0 0x7f6394198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 -- 172.21.15.64:0/2075470483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639419d2b0 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f639aae8640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394107fc0 0x7f639419cbd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f638c00cad0 tx=0x7f638c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.387+0000 7f638b7fe640 1 -- 172.21.15.64:0/2075470483 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c00ae20 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.388+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639419d5a0 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.388+0000 7f638b7fe640 1 -- 172.21.15.64:0/2075470483 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c005020 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.388+0000 7f638b7fe640 1 -- 172.21.15.64:0/2075470483 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c00f6f0 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.388+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6394100650 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.388+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f639406a520 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.390+0000 7f638b7fe640 1 -- 172.21.15.64:0/2075470483 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f638c0051d0 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.390+0000 7f638b7fe640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f63800a70a0 0x7f63800a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.390+0000 7f638b7fe640 1 -- 172.21.15.64:0/2075470483 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f638c0cf560 con 0x7f6394107fc0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.390+0000 7f639a2e7640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f63800a70a0 0x7f63800a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.391+0000 7f639a2e7640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f63800a70a0 0x7f63800a9560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6390009b60 tx=0x7f6390005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:23.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.395+0000 7f638b7fe640 1 -- 172.21.15.64:0/2075470483 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f638c014030 con 0x7f6394107fc0 2024-08-19T01:32:23.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.571+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f6394109fd0 con 0x7f6394107fc0 2024-08-19T01:32:23.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.571+0000 7f638b7fe640 1 -- 172.21.15.64:0/2075470483 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f638c09c000 con 0x7f6394107fc0 2024-08-19T01:32:23.574 INFO:teuthology.orchestra.run.smithi064.stdout:274877906985 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f63800a70a0 msgr2=0x7f63800a9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f63800a70a0 0x7f63800a9560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6390009b60 tx=0x7f6390005f90 comp rx=0 tx=0).stop 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394107fc0 msgr2=0x7f639419cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394107fc0 0x7f639419cbd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f638c00cad0 tx=0x7f638c00cf90 comp rx=0 tx=0).stop 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 shutdown_connections 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f63800a70a0 0x7f63800a9560 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6394107fc0 0x7f639419cbd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6394103470 0x7f6394198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 --2- 172.21.15.64:0/2075470483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6394102ac0 0x7f6394198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 >> 172.21.15.64:0/2075470483 conn(0x7f63940fc500 msgr2=0x7f6394105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 shutdown_connections 2024-08-19T01:32:23.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:23.573+0000 7f639c54b640 1 -- 172.21.15.64:0/2075470483 wait complete. 2024-08-19T01:32:23.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:23 smithi123 ceph-mon[21829]: pgmap v315: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:23.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:23 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/195090191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T01:32:23.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:23 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1039866076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T01:32:23.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:23 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2075470483' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T01:32:23.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:23 smithi161 ceph-mon[21757]: pgmap v315: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:23.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:23 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/195090191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T01:32:23.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:23 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1039866076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T01:32:23.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:23 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2075470483' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T01:32:23.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:23 smithi064 ceph-mon[18515]: pgmap v315: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-08-19T01:32:23.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:23 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/195090191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T01:32:23.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:23 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1039866076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T01:32:23.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:23 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2075470483' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T01:32:24.298 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574921 got 55834574923 for osd.1 2024-08-19T01:32:24.299 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:24.916 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822712 got 154618822715 for osd.5 2024-08-19T01:32:24.916 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:25.327 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481820 got 330712481822 for osd.11 2024-08-19T01:32:25.327 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:25.730 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593780 got 184683593782 for osd.6 2024-08-19T01:32:25.731 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:25.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:25 smithi123 ceph-mon[21829]: pgmap v316: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:25.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:25 smithi161 ceph-mon[21757]: pgmap v316: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:25.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:25 smithi064 ceph-mon[18515]: pgmap v316: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:26.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906982 got 274877906985 for osd.9 2024-08-19T01:32:26.224 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:26.719 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:26.751 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738445 got 34359738447 for osd.0 2024-08-19T01:32:26.751 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 -- 172.21.15.64:0/984447693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281018b0 msgr2=0x7ff028101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 --2- 172.21.15.64:0/984447693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281018b0 0x7ff028101d10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff01c009930 tx=0x7ff01c02a420 comp rx=0 tx=0).stop 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 -- 172.21.15.64:0/984447693 shutdown_connections 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 --2- 172.21.15.64:0/984447693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff028102250 0x7ff02810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 --2- 172.21.15.64:0/984447693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281018b0 0x7ff028101d10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 --2- 172.21.15.64:0/984447693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0281078b0 0x7ff028107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 -- 172.21.15.64:0/984447693 >> 172.21.15.64:0/984447693 conn(0x7ff0280fd5f0 msgr2=0x7ff0280ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 -- 172.21.15.64:0/984447693 shutdown_connections 2024-08-19T01:32:27.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.211+0000 7ff02e713640 1 -- 172.21.15.64:0/984447693 wait complete. 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 Processor -- start 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 -- start start 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0281018b0 0x7ff028198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff028102250 0x7ff0281986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281078b0 0x7ff02819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff028111cb0 con 0x7ff0281078b0 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff028111b30 con 0x7ff028102250 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02e713640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff028111e30 con 0x7ff0281018b0 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02ccb0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281078b0 0x7ff02819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02ccb0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281078b0 0x7ff02819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53364/0 (socket says 172.21.15.64:53364) 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff02ccb0640 1 -- 172.21.15.64:0/924605463 learned_addr learned my addr 172.21.15.64:0/924605463 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.212+0000 7ff0277fe640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff028102250 0x7ff0281986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff027fff640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0281018b0 0x7ff028198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02ccb0640 1 -- 172.21.15.64:0/924605463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0281018b0 msgr2=0x7ff028198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02ccb0640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0281018b0 0x7ff028198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02ccb0640 1 -- 172.21.15.64:0/924605463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff028102250 msgr2=0x7ff0281986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02ccb0640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff028102250 0x7ff0281986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02ccb0640 1 -- 172.21.15.64:0/924605463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff02819d120 con 0x7ff0281078b0 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff027fff640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0281018b0 0x7ff028198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02ccb0640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281078b0 0x7ff02819ca40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff02000cce0 tx=0x7ff02000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff0257fa640 1 -- 172.21.15.64:0/924605463 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02000f450 con 0x7ff0281078b0 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff0257fa640 1 -- 172.21.15.64:0/924605463 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff02000fad0 con 0x7ff0281078b0 2024-08-19T01:32:27.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff0257fa640 1 -- 172.21.15.64:0/924605463 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020018c10 con 0x7ff0281078b0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff02819d410 con 0x7ff0281078b0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.213+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0281a4cf0 con 0x7ff0281078b0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.214+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff028101d10 con 0x7ff0281078b0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.216+0000 7ff0257fa640 1 -- 172.21.15.64:0/924605463 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff02000f5f0 con 0x7ff0281078b0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.217+0000 7ff0257fa640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0100a2c90 0x7ff0100a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.217+0000 7ff027fff640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0100a2c90 0x7ff0100a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.217+0000 7ff0257fa640 1 -- 172.21.15.64:0/924605463 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff0200cfcc0 con 0x7ff0281078b0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.217+0000 7ff027fff640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0100a2c90 0x7ff0100a5150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff018005d70 tx=0x7ff018005d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:27.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.220+0000 7ff0257fa640 1 -- 172.21.15.64:0/924605463 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff020014030 con 0x7ff0281078b0 2024-08-19T01:32:27.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.396+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff028109890 con 0x7ff0281078b0 2024-08-19T01:32:27.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.396+0000 7ff0257fa640 1 -- 172.21.15.64:0/924605463 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff02009c870 con 0x7ff0281078b0 2024-08-19T01:32:27.399 INFO:teuthology.orchestra.run.smithi064.stdout:77309411401 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0100a2c90 msgr2=0x7ff0100a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0100a2c90 0x7ff0100a5150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff018005d70 tx=0x7ff018005d00 comp rx=0 tx=0).stop 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281078b0 msgr2=0x7ff02819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281078b0 0x7ff02819ca40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff02000cce0 tx=0x7ff02000a620 comp rx=0 tx=0).stop 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 shutdown_connections 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff0100a2c90 0x7ff0100a5150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0281078b0 0x7ff02819ca40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff028102250 0x7ff0281986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 --2- 172.21.15.64:0/924605463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0281018b0 0x7ff028198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:27.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.398+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 >> 172.21.15.64:0/924605463 conn(0x7ff0280fd5f0 msgr2=0x7ff0280fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:27.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.399+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 shutdown_connections 2024-08-19T01:32:27.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:27.399+0000 7ff02e713640 1 -- 172.21.15.64:0/924605463 wait complete. 2024-08-19T01:32:27.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:27 smithi064 ceph-mon[18515]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:27.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:27 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/924605463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T01:32:27.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:27 smithi123 ceph-mon[21829]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:27.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:27 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/924605463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T01:32:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:27 smithi161 ceph-mon[21757]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:27 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/924605463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T01:32:28.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411397 got 77309411401 for osd.2 2024-08-19T01:32:28.198 DEBUG:teuthology.parallel:result is None 2024-08-19T01:32:28.198 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-19T01:32:28.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T01:32:28.454 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:28.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 -- 172.21.15.64:0/361763244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 msgr2=0x7f479c102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 --2- 172.21.15.64:0/361763244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c102e30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f478c009900 tx=0x7f478c02a480 comp rx=0 tx=0).stop 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 -- 172.21.15.64:0/361763244 shutdown_connections 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 --2- 172.21.15.64:0/361763244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c10be80 0x7f479c10e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 --2- 172.21.15.64:0/361763244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f479c103370 0x7f479c10b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 --2- 172.21.15.64:0/361763244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c102e30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 -- 172.21.15.64:0/361763244 >> 172.21.15.64:0/361763244 conn(0x7f479c0fc4b0 msgr2=0x7f479c0fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 -- 172.21.15.64:0/361763244 shutdown_connections 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.950+0000 7f47a33fd640 1 -- 172.21.15.64:0/361763244 wait complete. 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 Processor -- start 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 -- start start 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c103370 0x7f479c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:28.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f479c10be80 0x7f479c19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f479c111cc0 con 0x7f479c102a50 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f479c111b40 con 0x7f479c103370 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a33fd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f479c111e40 con 0x7f479c10be80 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45436/0 (socket says 172.21.15.64:45436) 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 -- 172.21.15.64:0/2163178507 learned_addr learned my addr 172.21.15.64:0/2163178507 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a0998640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c103370 0x7f479c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a199a640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f479c10be80 0x7f479c19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 -- 172.21.15.64:0/2163178507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f479c10be80 msgr2=0x7f479c19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f479c10be80 0x7f479c19ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 -- 172.21.15.64:0/2163178507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c103370 msgr2=0x7f479c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:28.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c103370 0x7f479c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a1199640 1 -- 172.21.15.64:0/2163178507 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479c19d140 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.951+0000 7f47a0998640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c103370 0x7f479c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.952+0000 7f47a1199640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c198190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f478c005ab0 tx=0x7f478c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.952+0000 7f47927fc640 1 -- 172.21.15.64:0/2163178507 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c03f030 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.952+0000 7f47927fc640 1 -- 172.21.15.64:0/2163178507 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c003ce0 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.952+0000 7f47927fc640 1 -- 172.21.15.64:0/2163178507 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c035070 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.952+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479c19d3d0 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.952+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f479c1a4d10 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.953+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f479c06ada0 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.954+0000 7f47927fc640 1 -- 172.21.15.64:0/2163178507 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f478c042030 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.955+0000 7f47927fc640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47880a2c10 0x7f47880a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.955+0000 7f47a0998640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47880a2c10 0x7f47880a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.955+0000 7f47927fc640 1 -- 172.21.15.64:0/2163178507 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f478c0f0120 con 0x7f479c102a50 2024-08-19T01:32:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.955+0000 7f47a0998640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47880a2c10 0x7f47880a50d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f479c0feab0 tx=0x7f4798008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:28.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:28.958+0000 7f47927fc640 1 -- 172.21.15.64:0/2163178507 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f478c0bcbc0 con 0x7f479c102a50 2024-08-19T01:32:29.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.130+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f479c106f60 con 0x7f47880a2c10 2024-08-19T01:32:29.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.139+0000 7f47927fc640 1 -- 172.21.15.64:0/2163178507 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340636 (secure 0 0 0) 0x7f47980095c0 con 0x7f47880a2c10 2024-08-19T01:32:29.145 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:29.146 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-08-19T01:32:29.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47880a2c10 msgr2=0x7f47880a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:29.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47880a2c10 0x7f47880a50d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f479c0feab0 tx=0x7f4798008040 comp rx=0 tx=0).stop 2024-08-19T01:32:29.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 msgr2=0x7f479c198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:29.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c198190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f478c005ab0 tx=0x7f478c0042f0 comp rx=0 tx=0).stop 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 shutdown_connections 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f47880a2c10 0x7f47880a50d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f479c10be80 0x7f479c19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c103370 0x7f479c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 --2- 172.21.15.64:0/2163178507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f479c102a50 0x7f479c198190 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 >> 172.21.15.64:0/2163178507 conn(0x7f479c0fc4b0 msgr2=0x7f479c0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 shutdown_connections 2024-08-19T01:32:29.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:29.143+0000 7f47a33fd640 1 -- 172.21.15.64:0/2163178507 wait complete. 2024-08-19T01:32:29.827 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:29 smithi064 ceph-mon[18515]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:29.858 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-08-19T01:32:27.697163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657062,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":456,"num_read_kb":345,"num_write":351,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1185199,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361424,"kb_used_data":12248,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580400,"statfs":{"total":1151940427776,"available":1151570329600,"internally_reserved":0,"allocated":12541952,"data_stored":8108708,"data_compressed":390537,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003063"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595389+0000","last_change":"2024-08-19T01:30:49.539512+0000","last_active":"2024-08-19T01:31:35.595389+0000","last_peered":"2024-08-19T01:31:35.595389+0000","last_clean":"2024-08-19T01:31:35.595389+0000","last_became_active":"2024-08-19T01:30:49.539163+0000","last_became_peered":"2024-08-19T01:30:49.539163+0000","last_unstale":"2024-08-19T01:31:35.595389+0000","last_undegraded":"2024-08-19T01:31:35.595389+0000","last_fullsized":"2024-08-19T01:31:35.595389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:23.874916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868682+0000","last_change":"2024-08-19T01:30:51.561135+0000","last_active":"2024-08-19T01:31:35.868682+0000","last_peered":"2024-08-19T01:31:35.868682+0000","last_clean":"2024-08-19T01:31:35.868682+0000","last_became_active":"2024-08-19T01:30:51.560822+0000","last_became_peered":"2024-08-19T01:30:51.560822+0000","last_unstale":"2024-08-19T01:31:35.868682+0000","last_undegraded":"2024-08-19T01:31:35.868682+0000","last_fullsized":"2024-08-19T01:31:35.868682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:08:35.918895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597319+0000","last_change":"2024-08-19T01:30:58.595727+0000","last_active":"2024-08-19T01:31:35.597319+0000","last_peered":"2024-08-19T01:31:35.597319+0000","last_clean":"2024-08-19T01:31:35.597319+0000","last_became_active":"2024-08-19T01:30:58.595286+0000","last_became_peered":"2024-08-19T01:30:58.595286+0000","last_unstale":"2024-08-19T01:31:35.597319+0000","last_undegraded":"2024-08-19T01:31:35.597319+0000","last_fullsized":"2024-08-19T01:31:35.597319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:25:02.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868994+0000","last_change":"2024-08-19T01:30:49.575693+0000","last_active":"2024-08-19T01:31:35.868994+0000","last_peered":"2024-08-19T01:31:35.868994+0000","last_clean":"2024-08-19T01:31:35.868994+0000","last_became_active":"2024-08-19T01:30:49.575331+0000","last_became_peered":"2024-08-19T01:30:49.575331+0000","last_unstale":"2024-08-19T01:31:35.868994+0000","last_undegraded":"2024-08-19T01:31:35.868994+0000","last_fullsized":"2024-08-19T01:31:35.868994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:31:50.773160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598318+0000","last_change":"2024-08-19T01:30:51.562446+0000","last_active":"2024-08-19T01:31:35.598318+0000","last_peered":"2024-08-19T01:31:35.598318+0000","last_clean":"2024-08-19T01:31:35.598318+0000","last_became_active":"2024-08-19T01:30:51.561475+0000","last_became_peered":"2024-08-19T01:30:51.561475+0000","last_unstale":"2024-08-19T01:31:35.598318+0000","last_undegraded":"2024-08-19T01:31:35.598318+0000","last_fullsized":"2024-08-19T01:31:35.598318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:51:41.531247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599871+0000","last_change":"2024-08-19T01:30:58.591371+0000","last_active":"2024-08-19T01:31:35.599871+0000","last_peered":"2024-08-19T01:31:35.599871+0000","last_clean":"2024-08-19T01:31:35.599871+0000","last_became_active":"2024-08-19T01:30:58.590936+0000","last_became_peered":"2024-08-19T01:30:58.590936+0000","last_unstale":"2024-08-19T01:31:35.599871+0000","last_undegraded":"2024-08-19T01:31:35.599871+0000","last_fullsized":"2024-08-19T01:31:35.599871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:07:33.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.613588+0000","last_change":"2024-08-19T01:31:35.613145+0000","last_active":"2024-08-19T01:31:35.613588+0000","last_peered":"2024-08-19T01:31:35.613588+0000","last_clean":"2024-08-19T01:31:35.613588+0000","last_became_active":"2024-08-19T01:31:35.605087+0000","last_became_peered":"2024-08-19T01:31:35.605087+0000","last_unstale":"2024-08-19T01:31:35.613588+0000","last_undegraded":"2024-08-19T01:31:35.613588+0000","last_fullsized":"2024-08-19T01:31:35.613588+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:25:59.109707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596225+0000","last_change":"2024-08-19T01:30:51.555256+0000","last_active":"2024-08-19T01:31:35.596225+0000","last_peered":"2024-08-19T01:31:35.596225+0000","last_clean":"2024-08-19T01:31:35.596225+0000","last_became_active":"2024-08-19T01:30:51.554030+0000","last_became_peered":"2024-08-19T01:30:51.554030+0000","last_unstale":"2024-08-19T01:31:35.596225+0000","last_undegraded":"2024-08-19T01:31:35.596225+0000","last_fullsized":"2024-08-19T01:31:35.596225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:12:04.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600498+0000","last_change":"2024-08-19T01:30:58.589362+0000","last_active":"2024-08-19T01:31:35.600498+0000","last_peered":"2024-08-19T01:31:35.600498+0000","last_clean":"2024-08-19T01:31:35.600498+0000","last_became_active":"2024-08-19T01:30:58.588937+0000","last_became_peered":"2024-08-19T01:30:58.588937+0000","last_unstale":"2024-08-19T01:31:35.600498+0000","last_undegraded":"2024-08-19T01:31:35.600498+0000","last_fullsized":"2024-08-19T01:31:35.600498+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:31.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604267+0000","last_change":"2024-08-19T01:30:49.538597+0000","last_active":"2024-08-19T01:31:35.604267+0000","last_peered":"2024-08-19T01:31:35.604267+0000","last_clean":"2024-08-19T01:31:35.604267+0000","last_became_active":"2024-08-19T01:30:49.538228+0000","last_became_peered":"2024-08-19T01:30:49.538228+0000","last_unstale":"2024-08-19T01:31:35.604267+0000","last_undegraded":"2024-08-19T01:31:35.604267+0000","last_fullsized":"2024-08-19T01:31:35.604267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:31.350962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600891+0000","last_change":"2024-08-19T01:30:51.562829+0000","last_active":"2024-08-19T01:31:35.600891+0000","last_peered":"2024-08-19T01:31:35.600891+0000","last_clean":"2024-08-19T01:31:35.600891+0000","last_became_active":"2024-08-19T01:30:51.562144+0000","last_became_peered":"2024-08-19T01:30:51.562144+0000","last_unstale":"2024-08-19T01:31:35.600891+0000","last_undegraded":"2024-08-19T01:31:35.600891+0000","last_fullsized":"2024-08-19T01:31:35.600891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:17:09.184222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599645+0000","last_change":"2024-08-19T01:30:58.593178+0000","last_active":"2024-08-19T01:31:35.599645+0000","last_peered":"2024-08-19T01:31:35.599645+0000","last_clean":"2024-08-19T01:31:35.599645+0000","last_became_active":"2024-08-19T01:30:58.591247+0000","last_became_peered":"2024-08-19T01:30:58.591247+0000","last_unstale":"2024-08-19T01:31:35.599645+0000","last_undegraded":"2024-08-19T01:31:35.599645+0000","last_fullsized":"2024-08-19T01:31:35.599645+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:20:40.796135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604428+0000","last_change":"2024-08-19T01:30:51.562307+0000","last_active":"2024-08-19T01:31:35.604428+0000","last_peered":"2024-08-19T01:31:35.604428+0000","last_clean":"2024-08-19T01:31:35.604428+0000","last_became_active":"2024-08-19T01:30:51.561957+0000","last_became_peered":"2024-08-19T01:30:51.561957+0000","last_unstale":"2024-08-19T01:31:35.604428+0000","last_undegraded":"2024-08-19T01:31:35.604428+0000","last_fullsized":"2024-08-19T01:31:35.604428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:22.902317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.610093+0000","last_change":"2024-08-19T01:31:35.609612+0000","last_active":"2024-08-19T01:31:35.610093+0000","last_peered":"2024-08-19T01:31:35.610093+0000","last_clean":"2024-08-19T01:31:35.610093+0000","last_became_active":"2024-08-19T01:31:35.604969+0000","last_became_peered":"2024-08-19T01:31:35.604969+0000","last_unstale":"2024-08-19T01:31:35.610093+0000","last_undegraded":"2024-08-19T01:31:35.610093+0000","last_fullsized":"2024-08-19T01:31:35.610093+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:59:38.089119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600108+0000","last_change":"2024-08-19T01:30:58.588984+0000","last_active":"2024-08-19T01:31:35.600108+0000","last_peered":"2024-08-19T01:31:35.600108+0000","last_clean":"2024-08-19T01:31:35.600108+0000","last_became_active":"2024-08-19T01:30:58.588504+0000","last_became_peered":"2024-08-19T01:30:58.588504+0000","last_unstale":"2024-08-19T01:31:35.600108+0000","last_undegraded":"2024-08-19T01:31:35.600108+0000","last_fullsized":"2024-08-19T01:31:35.600108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:29:14.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869072+0000","last_change":"2024-08-19T01:30:51.553293+0000","last_active":"2024-08-19T01:31:35.869072+0000","last_peered":"2024-08-19T01:31:35.869072+0000","last_clean":"2024-08-19T01:31:35.869072+0000","last_became_active":"2024-08-19T01:30:51.552990+0000","last_became_peered":"2024-08-19T01:30:51.552990+0000","last_unstale":"2024-08-19T01:31:35.869072+0000","last_undegraded":"2024-08-19T01:31:35.869072+0000","last_fullsized":"2024-08-19T01:31:35.869072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:28:33.193014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597523+0000","last_change":"2024-08-19T01:30:49.537742+0000","last_active":"2024-08-19T01:31:35.597523+0000","last_peered":"2024-08-19T01:31:35.597523+0000","last_clean":"2024-08-19T01:31:35.597523+0000","last_became_active":"2024-08-19T01:30:49.537153+0000","last_became_peered":"2024-08-19T01:30:49.537153+0000","last_unstale":"2024-08-19T01:31:35.597523+0000","last_undegraded":"2024-08-19T01:31:35.597523+0000","last_fullsized":"2024-08-19T01:31:35.597523+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:25:03.650900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600493+0000","last_change":"2024-08-19T01:30:58.592348+0000","last_active":"2024-08-19T01:31:35.600493+0000","last_peered":"2024-08-19T01:31:35.600493+0000","last_clean":"2024-08-19T01:31:35.600493+0000","last_became_active":"2024-08-19T01:30:58.591641+0000","last_became_peered":"2024-08-19T01:30:58.591641+0000","last_unstale":"2024-08-19T01:31:35.600493+0000","last_undegraded":"2024-08-19T01:31:35.600493+0000","last_fullsized":"2024-08-19T01:31:35.600493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:39:17.542800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599683+0000","last_change":"2024-08-19T01:30:51.552126+0000","last_active":"2024-08-19T01:31:35.599683+0000","last_peered":"2024-08-19T01:31:35.599683+0000","last_clean":"2024-08-19T01:31:35.599683+0000","last_became_active":"2024-08-19T01:30:51.551795+0000","last_became_peered":"2024-08-19T01:30:51.551795+0000","last_unstale":"2024-08-19T01:31:35.599683+0000","last_undegraded":"2024-08-19T01:31:35.599683+0000","last_fullsized":"2024-08-19T01:31:35.599683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:22:54.705359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868723+0000","last_change":"2024-08-19T01:30:49.536659+0000","last_active":"2024-08-19T01:31:35.868723+0000","last_peered":"2024-08-19T01:31:35.868723+0000","last_clean":"2024-08-19T01:31:35.868723+0000","last_became_active":"2024-08-19T01:30:49.536366+0000","last_became_peered":"2024-08-19T01:30:49.536366+0000","last_unstale":"2024-08-19T01:31:35.868723+0000","last_undegraded":"2024-08-19T01:31:35.868723+0000","last_fullsized":"2024-08-19T01:31:35.868723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:58:52.547461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595100+0000","last_change":"2024-08-19T01:30:58.593724+0000","last_active":"2024-08-19T01:31:35.595100+0000","last_peered":"2024-08-19T01:31:35.595100+0000","last_clean":"2024-08-19T01:31:35.595100+0000","last_became_active":"2024-08-19T01:30:58.593383+0000","last_became_peered":"2024-08-19T01:30:58.593383+0000","last_unstale":"2024-08-19T01:31:35.595100+0000","last_undegraded":"2024-08-19T01:31:35.595100+0000","last_fullsized":"2024-08-19T01:31:35.595100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:54:26.424139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603836+0000","last_change":"2024-08-19T01:30:51.557682+0000","last_active":"2024-08-19T01:31:35.603836+0000","last_peered":"2024-08-19T01:31:35.603836+0000","last_clean":"2024-08-19T01:31:35.603836+0000","last_became_active":"2024-08-19T01:30:51.557351+0000","last_became_peered":"2024-08-19T01:30:51.557351+0000","last_unstale":"2024-08-19T01:31:35.603836+0000","last_undegraded":"2024-08-19T01:31:35.603836+0000","last_fullsized":"2024-08-19T01:31:35.603836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:41.484955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595958+0000","last_change":"2024-08-19T01:30:49.575996+0000","last_active":"2024-08-19T01:31:35.595958+0000","last_peered":"2024-08-19T01:31:35.595958+0000","last_clean":"2024-08-19T01:31:35.595958+0000","last_became_active":"2024-08-19T01:30:49.575607+0000","last_became_peered":"2024-08-19T01:30:49.575607+0000","last_unstale":"2024-08-19T01:31:35.595958+0000","last_undegraded":"2024-08-19T01:31:35.595958+0000","last_fullsized":"2024-08-19T01:31:35.595958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:20:33.467100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600113+0000","last_change":"2024-08-19T01:30:58.592629+0000","last_active":"2024-08-19T01:31:35.600113+0000","last_peered":"2024-08-19T01:31:35.600113+0000","last_clean":"2024-08-19T01:31:35.600113+0000","last_became_active":"2024-08-19T01:30:58.592122+0000","last_became_peered":"2024-08-19T01:30:58.592122+0000","last_unstale":"2024-08-19T01:31:35.600113+0000","last_undegraded":"2024-08-19T01:31:35.600113+0000","last_fullsized":"2024-08-19T01:31:35.600113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:19:09.846829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.746416+0000","last_change":"2024-08-19T01:31:35.746416+0000","last_active":"2024-08-19T01:31:35.746416+0000","last_peered":"2024-08-19T01:31:35.746416+0000","last_clean":"2024-08-19T01:31:35.746416+0000","last_became_active":"2024-08-19T01:31:35.746110+0000","last_became_peered":"2024-08-19T01:31:35.746110+0000","last_unstale":"2024-08-19T01:31:35.746416+0000","last_undegraded":"2024-08-19T01:31:35.746416+0000","last_fullsized":"2024-08-19T01:31:35.746416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:29:29.789785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600943+0000","last_change":"2024-08-19T01:30:49.711366+0000","last_active":"2024-08-19T01:31:35.600943+0000","last_peered":"2024-08-19T01:31:35.600943+0000","last_clean":"2024-08-19T01:31:35.600943+0000","last_became_active":"2024-08-19T01:30:49.711078+0000","last_became_peered":"2024-08-19T01:30:49.711078+0000","last_unstale":"2024-08-19T01:31:35.600943+0000","last_undegraded":"2024-08-19T01:31:35.600943+0000","last_fullsized":"2024-08-19T01:31:35.600943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:11:16.767867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597334+0000","last_change":"2024-08-19T01:30:58.591845+0000","last_active":"2024-08-19T01:31:35.597334+0000","last_peered":"2024-08-19T01:31:35.597334+0000","last_clean":"2024-08-19T01:31:35.597334+0000","last_became_active":"2024-08-19T01:30:58.591453+0000","last_became_peered":"2024-08-19T01:30:58.591453+0000","last_unstale":"2024-08-19T01:31:35.597334+0000","last_undegraded":"2024-08-19T01:31:35.597334+0000","last_fullsized":"2024-08-19T01:31:35.597334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:37:09.521514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869013+0000","last_change":"2024-08-19T01:30:51.563156+0000","last_active":"2024-08-19T01:31:35.869013+0000","last_peered":"2024-08-19T01:31:35.869013+0000","last_clean":"2024-08-19T01:31:35.869013+0000","last_became_active":"2024-08-19T01:30:51.562860+0000","last_became_peered":"2024-08-19T01:30:51.562860+0000","last_unstale":"2024-08-19T01:31:35.869013+0000","last_undegraded":"2024-08-19T01:31:35.869013+0000","last_fullsized":"2024-08-19T01:31:35.869013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:54:11.768833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:48.559439+0000","last_change":"2024-08-19T01:30:49.536648+0000","last_active":"2024-08-19T01:31:48.559439+0000","last_peered":"2024-08-19T01:31:48.559439+0000","last_clean":"2024-08-19T01:31:48.559439+0000","last_became_active":"2024-08-19T01:30:49.536080+0000","last_became_peered":"2024-08-19T01:30:49.536080+0000","last_unstale":"2024-08-19T01:31:48.559439+0000","last_undegraded":"2024-08-19T01:31:48.559439+0000","last_fullsized":"2024-08-19T01:31:48.559439+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:03:46.804252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868291+0000","last_change":"2024-08-19T01:30:58.592253+0000","last_active":"2024-08-19T01:31:35.868291+0000","last_peered":"2024-08-19T01:31:35.868291+0000","last_clean":"2024-08-19T01:31:35.868291+0000","last_became_active":"2024-08-19T01:30:58.591911+0000","last_became_peered":"2024-08-19T01:30:58.591911+0000","last_unstale":"2024-08-19T01:31:35.868291+0000","last_undegraded":"2024-08-19T01:31:35.868291+0000","last_fullsized":"2024-08-19T01:31:35.868291+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:02:27.112597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745492+0000","last_change":"2024-08-19T01:30:51.547607+0000","last_active":"2024-08-19T01:31:35.745492+0000","last_peered":"2024-08-19T01:31:35.745492+0000","last_clean":"2024-08-19T01:31:35.745492+0000","last_became_active":"2024-08-19T01:30:51.547316+0000","last_became_peered":"2024-08-19T01:30:51.547316+0000","last_unstale":"2024-08-19T01:31:35.745492+0000","last_undegraded":"2024-08-19T01:31:35.745492+0000","last_fullsized":"2024-08-19T01:31:35.745492+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:48:58.300508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604727+0000","last_change":"2024-08-19T01:30:49.576942+0000","last_active":"2024-08-19T01:31:35.604727+0000","last_peered":"2024-08-19T01:31:35.604727+0000","last_clean":"2024-08-19T01:31:35.604727+0000","last_became_active":"2024-08-19T01:30:49.576528+0000","last_became_peered":"2024-08-19T01:30:49.576528+0000","last_unstale":"2024-08-19T01:31:35.604727+0000","last_undegraded":"2024-08-19T01:31:35.604727+0000","last_fullsized":"2024-08-19T01:31:35.604727+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:23:53.259395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600274+0000","last_change":"2024-08-19T01:30:58.591577+0000","last_active":"2024-08-19T01:31:35.600274+0000","last_peered":"2024-08-19T01:31:35.600274+0000","last_clean":"2024-08-19T01:31:35.600274+0000","last_became_active":"2024-08-19T01:30:58.591181+0000","last_became_peered":"2024-08-19T01:30:58.591181+0000","last_unstale":"2024-08-19T01:31:35.600274+0000","last_undegraded":"2024-08-19T01:31:35.600274+0000","last_fullsized":"2024-08-19T01:31:35.600274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:02:59.682147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598931+0000","last_change":"2024-08-19T01:30:51.548556+0000","last_active":"2024-08-19T01:31:35.598931+0000","last_peered":"2024-08-19T01:31:35.598931+0000","last_clean":"2024-08-19T01:31:35.598931+0000","last_became_active":"2024-08-19T01:30:51.547838+0000","last_became_peered":"2024-08-19T01:30:51.547838+0000","last_unstale":"2024-08-19T01:31:35.598931+0000","last_undegraded":"2024-08-19T01:31:35.598931+0000","last_fullsized":"2024-08-19T01:31:35.598931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:08:49.671964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595172+0000","last_change":"2024-08-19T01:30:49.576799+0000","last_active":"2024-08-19T01:31:35.595172+0000","last_peered":"2024-08-19T01:31:35.595172+0000","last_clean":"2024-08-19T01:31:35.595172+0000","last_became_active":"2024-08-19T01:30:49.576385+0000","last_became_peered":"2024-08-19T01:30:49.576385+0000","last_unstale":"2024-08-19T01:31:35.595172+0000","last_undegraded":"2024-08-19T01:31:35.595172+0000","last_fullsized":"2024-08-19T01:31:35.595172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:34:18.287518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603305+0000","last_change":"2024-08-19T01:30:58.593703+0000","last_active":"2024-08-19T01:31:35.603305+0000","last_peered":"2024-08-19T01:31:35.603305+0000","last_clean":"2024-08-19T01:31:35.603305+0000","last_became_active":"2024-08-19T01:30:58.593303+0000","last_became_peered":"2024-08-19T01:30:58.593303+0000","last_unstale":"2024-08-19T01:31:35.603305+0000","last_undegraded":"2024-08-19T01:31:35.603305+0000","last_fullsized":"2024-08-19T01:31:35.603305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:12:43.359262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601173+0000","last_change":"2024-08-19T01:31:35.601173+0000","last_active":"2024-08-19T01:31:35.601173+0000","last_peered":"2024-08-19T01:31:35.601173+0000","last_clean":"2024-08-19T01:31:35.601173+0000","last_became_active":"2024-08-19T01:31:35.600742+0000","last_became_peered":"2024-08-19T01:31:35.600742+0000","last_unstale":"2024-08-19T01:31:35.601173+0000","last_undegraded":"2024-08-19T01:31:35.601173+0000","last_fullsized":"2024-08-19T01:31:35.601173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:36:48.977124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604573+0000","last_change":"2024-08-19T01:30:49.540070+0000","last_active":"2024-08-19T01:31:35.604573+0000","last_peered":"2024-08-19T01:31:35.604573+0000","last_clean":"2024-08-19T01:31:35.604573+0000","last_became_active":"2024-08-19T01:30:49.539706+0000","last_became_peered":"2024-08-19T01:30:49.539706+0000","last_unstale":"2024-08-19T01:31:35.604573+0000","last_undegraded":"2024-08-19T01:31:35.604573+0000","last_fullsized":"2024-08-19T01:31:35.604573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:14:15.813670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745319+0000","last_change":"2024-08-19T01:30:58.590766+0000","last_active":"2024-08-19T01:31:35.745319+0000","last_peered":"2024-08-19T01:31:35.745319+0000","last_clean":"2024-08-19T01:31:35.745319+0000","last_became_active":"2024-08-19T01:30:58.590415+0000","last_became_peered":"2024-08-19T01:30:58.590415+0000","last_unstale":"2024-08-19T01:31:35.745319+0000","last_undegraded":"2024-08-19T01:31:35.745319+0000","last_fullsized":"2024-08-19T01:31:35.745319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:48:59.261086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.602761+0000","last_change":"2024-08-19T01:31:35.602332+0000","last_active":"2024-08-19T01:31:35.602761+0000","last_peered":"2024-08-19T01:31:35.602761+0000","last_clean":"2024-08-19T01:31:35.602761+0000","last_became_active":"2024-08-19T01:31:35.602007+0000","last_became_peered":"2024-08-19T01:31:35.602007+0000","last_unstale":"2024-08-19T01:31:35.602761+0000","last_undegraded":"2024-08-19T01:31:35.602761+0000","last_fullsized":"2024-08-19T01:31:35.602761+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:17:47.694567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:23.671848+0000","last_change":"2024-08-19T01:30:49.575995+0000","last_active":"2024-08-19T01:32:23.671848+0000","last_peered":"2024-08-19T01:32:23.671848+0000","last_clean":"2024-08-19T01:32:23.671848+0000","last_became_active":"2024-08-19T01:30:49.575249+0000","last_became_peered":"2024-08-19T01:30:49.575249+0000","last_unstale":"2024-08-19T01:32:23.671848+0000","last_undegraded":"2024-08-19T01:32:23.671848+0000","last_fullsized":"2024-08-19T01:32:23.671848+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:39:05.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745536+0000","last_change":"2024-08-19T01:30:58.598812+0000","last_active":"2024-08-19T01:31:35.745536+0000","last_peered":"2024-08-19T01:31:35.745536+0000","last_clean":"2024-08-19T01:31:35.745536+0000","last_became_active":"2024-08-19T01:30:58.598434+0000","last_became_peered":"2024-08-19T01:30:58.598434+0000","last_unstale":"2024-08-19T01:31:35.745536+0000","last_undegraded":"2024-08-19T01:31:35.745536+0000","last_fullsized":"2024-08-19T01:31:35.745536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:15:19.381503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744921+0000","last_change":"2024-08-19T01:30:51.549158+0000","last_active":"2024-08-19T01:31:35.744921+0000","last_peered":"2024-08-19T01:31:35.744921+0000","last_clean":"2024-08-19T01:31:35.744921+0000","last_became_active":"2024-08-19T01:30:51.548854+0000","last_became_peered":"2024-08-19T01:30:51.548854+0000","last_unstale":"2024-08-19T01:31:35.744921+0000","last_undegraded":"2024-08-19T01:31:35.744921+0000","last_fullsized":"2024-08-19T01:31:35.744921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:35.661758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604219+0000","last_change":"2024-08-19T01:30:49.538282+0000","last_active":"2024-08-19T01:31:35.604219+0000","last_peered":"2024-08-19T01:31:35.604219+0000","last_clean":"2024-08-19T01:31:35.604219+0000","last_became_active":"2024-08-19T01:30:49.537918+0000","last_became_peered":"2024-08-19T01:30:49.537918+0000","last_unstale":"2024-08-19T01:31:35.604219+0000","last_undegraded":"2024-08-19T01:31:35.604219+0000","last_fullsized":"2024-08-19T01:31:35.604219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:27.886958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601012+0000","last_change":"2024-08-19T01:30:58.591153+0000","last_active":"2024-08-19T01:31:35.601012+0000","last_peered":"2024-08-19T01:31:35.601012+0000","last_clean":"2024-08-19T01:31:35.601012+0000","last_became_active":"2024-08-19T01:30:58.590738+0000","last_became_peered":"2024-08-19T01:30:58.590738+0000","last_unstale":"2024-08-19T01:31:35.601012+0000","last_undegraded":"2024-08-19T01:31:35.601012+0000","last_fullsized":"2024-08-19T01:31:35.601012+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:29.073825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868743+0000","last_change":"2024-08-19T01:30:51.554878+0000","last_active":"2024-08-19T01:31:35.868743+0000","last_peered":"2024-08-19T01:31:35.868743+0000","last_clean":"2024-08-19T01:31:35.868743+0000","last_became_active":"2024-08-19T01:30:51.553743+0000","last_became_peered":"2024-08-19T01:30:51.553743+0000","last_unstale":"2024-08-19T01:31:35.868743+0000","last_undegraded":"2024-08-19T01:31:35.868743+0000","last_fullsized":"2024-08-19T01:31:35.868743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:08:41.932641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600867+0000","last_change":"2024-08-19T01:30:49.542043+0000","last_active":"2024-08-19T01:31:35.600867+0000","last_peered":"2024-08-19T01:31:35.600867+0000","last_clean":"2024-08-19T01:31:35.600867+0000","last_became_active":"2024-08-19T01:30:49.541551+0000","last_became_peered":"2024-08-19T01:30:49.541551+0000","last_unstale":"2024-08-19T01:31:35.600867+0000","last_undegraded":"2024-08-19T01:31:35.600867+0000","last_fullsized":"2024-08-19T01:31:35.600867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:46:14.781516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":364,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604258+0000","last_change":"2024-08-19T01:29:41.574391+0000","last_active":"2024-08-19T01:31:35.604258+0000","last_peered":"2024-08-19T01:31:35.604258+0000","last_clean":"2024-08-19T01:31:35.604258+0000","last_became_active":"2024-08-19T01:29:41.573885+0000","last_became_peered":"2024-08-19T01:29:41.573885+0000","last_unstale":"2024-08-19T01:31:35.604258+0000","last_undegraded":"2024-08-19T01:31:35.604258+0000","last_fullsized":"2024-08-19T01:31:35.604258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_clean_scrub_stamp":"2024-08-19T01:26:37.926225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:55:38.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:58.598415+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:58.598086+0000","last_became_peered":"2024-08-19T01:30:58.598086+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:21:37.338869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605022+0000","last_change":"2024-08-19T01:30:51.560407+0000","last_active":"2024-08-19T01:31:35.605022+0000","last_peered":"2024-08-19T01:31:35.605022+0000","last_clean":"2024-08-19T01:31:35.605022+0000","last_became_active":"2024-08-19T01:30:51.559881+0000","last_became_peered":"2024-08-19T01:30:51.559881+0000","last_unstale":"2024-08-19T01:31:35.605022+0000","last_undegraded":"2024-08-19T01:31:35.605022+0000","last_fullsized":"2024-08-19T01:31:35.605022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:24:37.928254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597862+0000","last_change":"2024-08-19T01:30:49.711171+0000","last_active":"2024-08-19T01:31:35.597862+0000","last_peered":"2024-08-19T01:31:35.597862+0000","last_clean":"2024-08-19T01:31:35.597862+0000","last_became_active":"2024-08-19T01:30:49.710707+0000","last_became_peered":"2024-08-19T01:30:49.710707+0000","last_unstale":"2024-08-19T01:31:35.597862+0000","last_undegraded":"2024-08-19T01:31:35.597862+0000","last_fullsized":"2024-08-19T01:31:35.597862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:58:20.924502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595955+0000","last_change":"2024-08-19T01:30:58.590829+0000","last_active":"2024-08-19T01:31:35.595955+0000","last_peered":"2024-08-19T01:31:35.595955+0000","last_clean":"2024-08-19T01:31:35.595955+0000","last_became_active":"2024-08-19T01:30:58.590460+0000","last_became_peered":"2024-08-19T01:30:58.590460+0000","last_unstale":"2024-08-19T01:31:35.595955+0000","last_undegraded":"2024-08-19T01:31:35.595955+0000","last_fullsized":"2024-08-19T01:31:35.595955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:04.118550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604645+0000","last_change":"2024-08-19T01:30:51.564990+0000","last_active":"2024-08-19T01:31:35.604645+0000","last_peered":"2024-08-19T01:31:35.604645+0000","last_clean":"2024-08-19T01:31:35.604645+0000","last_became_active":"2024-08-19T01:30:51.564653+0000","last_became_peered":"2024-08-19T01:30:51.564653+0000","last_unstale":"2024-08-19T01:31:35.604645+0000","last_undegraded":"2024-08-19T01:31:35.604645+0000","last_fullsized":"2024-08-19T01:31:35.604645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:15:41.154376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601128+0000","last_change":"2024-08-19T01:30:49.537637+0000","last_active":"2024-08-19T01:31:35.601128+0000","last_peered":"2024-08-19T01:31:35.601128+0000","last_clean":"2024-08-19T01:31:35.601128+0000","last_became_active":"2024-08-19T01:30:49.537277+0000","last_became_peered":"2024-08-19T01:30:49.537277+0000","last_unstale":"2024-08-19T01:31:35.601128+0000","last_undegraded":"2024-08-19T01:31:35.601128+0000","last_fullsized":"2024-08-19T01:31:35.601128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:13.547597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598003+0000","last_change":"2024-08-19T01:30:58.594675+0000","last_active":"2024-08-19T01:31:35.598003+0000","last_peered":"2024-08-19T01:31:35.598003+0000","last_clean":"2024-08-19T01:31:35.598003+0000","last_became_active":"2024-08-19T01:30:58.594323+0000","last_became_peered":"2024-08-19T01:30:58.594323+0000","last_unstale":"2024-08-19T01:31:35.598003+0000","last_undegraded":"2024-08-19T01:31:35.598003+0000","last_fullsized":"2024-08-19T01:31:35.598003+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:21:05.447106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605468+0000","last_change":"2024-08-19T01:30:51.565014+0000","last_active":"2024-08-19T01:31:35.605468+0000","last_peered":"2024-08-19T01:31:35.605468+0000","last_clean":"2024-08-19T01:31:35.605468+0000","last_became_active":"2024-08-19T01:30:51.564672+0000","last_became_peered":"2024-08-19T01:30:51.564672+0000","last_unstale":"2024-08-19T01:31:35.605468+0000","last_undegraded":"2024-08-19T01:31:35.605468+0000","last_fullsized":"2024-08-19T01:31:35.605468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:56:07.404042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745571+0000","last_change":"2024-08-19T01:30:49.538423+0000","last_active":"2024-08-19T01:31:35.745571+0000","last_peered":"2024-08-19T01:31:35.745571+0000","last_clean":"2024-08-19T01:31:35.745571+0000","last_became_active":"2024-08-19T01:30:49.538049+0000","last_became_peered":"2024-08-19T01:30:49.538049+0000","last_unstale":"2024-08-19T01:31:35.745571+0000","last_undegraded":"2024-08-19T01:31:35.745571+0000","last_fullsized":"2024-08-19T01:31:35.745571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:27:41.449742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599789+0000","last_change":"2024-08-19T01:30:58.590963+0000","last_active":"2024-08-19T01:31:35.599789+0000","last_peered":"2024-08-19T01:31:35.599789+0000","last_clean":"2024-08-19T01:31:35.599789+0000","last_became_active":"2024-08-19T01:30:58.590584+0000","last_became_peered":"2024-08-19T01:30:58.590584+0000","last_unstale":"2024-08-19T01:31:35.599789+0000","last_undegraded":"2024-08-19T01:31:35.599789+0000","last_fullsized":"2024-08-19T01:31:35.599789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:37.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599809+0000","last_change":"2024-08-19T01:30:51.561298+0000","last_active":"2024-08-19T01:31:35.599809+0000","last_peered":"2024-08-19T01:31:35.599809+0000","last_clean":"2024-08-19T01:31:35.599809+0000","last_became_active":"2024-08-19T01:30:51.560919+0000","last_became_peered":"2024-08-19T01:30:51.560919+0000","last_unstale":"2024-08-19T01:31:35.599809+0000","last_undegraded":"2024-08-19T01:31:35.599809+0000","last_fullsized":"2024-08-19T01:31:35.599809+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:04:39.808838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604897+0000","last_change":"2024-08-19T01:30:49.541968+0000","last_active":"2024-08-19T01:31:35.604897+0000","last_peered":"2024-08-19T01:31:35.604897+0000","last_clean":"2024-08-19T01:31:35.604897+0000","last_became_active":"2024-08-19T01:30:49.541512+0000","last_became_peered":"2024-08-19T01:30:49.541512+0000","last_unstale":"2024-08-19T01:31:35.604897+0000","last_undegraded":"2024-08-19T01:31:35.604897+0000","last_fullsized":"2024-08-19T01:31:35.604897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:34:28.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600022+0000","last_change":"2024-08-19T01:30:58.594776+0000","last_active":"2024-08-19T01:31:35.600022+0000","last_peered":"2024-08-19T01:31:35.600022+0000","last_clean":"2024-08-19T01:31:35.600022+0000","last_became_active":"2024-08-19T01:30:58.594414+0000","last_became_peered":"2024-08-19T01:30:58.594414+0000","last_unstale":"2024-08-19T01:31:35.600022+0000","last_undegraded":"2024-08-19T01:31:35.600022+0000","last_fullsized":"2024-08-19T01:31:35.600022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:32:20.121504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600072+0000","last_change":"2024-08-19T01:30:51.556696+0000","last_active":"2024-08-19T01:31:35.600072+0000","last_peered":"2024-08-19T01:31:35.600072+0000","last_clean":"2024-08-19T01:31:35.600072+0000","last_became_active":"2024-08-19T01:30:51.555873+0000","last_became_peered":"2024-08-19T01:30:51.555873+0000","last_unstale":"2024-08-19T01:31:35.600072+0000","last_undegraded":"2024-08-19T01:31:35.600072+0000","last_fullsized":"2024-08-19T01:31:35.600072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:50:27.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603398+0000","last_change":"2024-08-19T01:30:49.576925+0000","last_active":"2024-08-19T01:31:35.603398+0000","last_peered":"2024-08-19T01:31:35.603398+0000","last_clean":"2024-08-19T01:31:35.603398+0000","last_became_active":"2024-08-19T01:30:49.576509+0000","last_became_peered":"2024-08-19T01:30:49.576509+0000","last_unstale":"2024-08-19T01:31:35.603398+0000","last_undegraded":"2024-08-19T01:31:35.603398+0000","last_fullsized":"2024-08-19T01:31:35.603398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:43:01.887020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745511+0000","last_change":"2024-08-19T01:30:58.590940+0000","last_active":"2024-08-19T01:31:35.745511+0000","last_peered":"2024-08-19T01:31:35.745511+0000","last_clean":"2024-08-19T01:31:35.745511+0000","last_became_active":"2024-08-19T01:30:58.590565+0000","last_became_peered":"2024-08-19T01:30:58.590565+0000","last_unstale":"2024-08-19T01:31:35.745511+0000","last_undegraded":"2024-08-19T01:31:35.745511+0000","last_fullsized":"2024-08-19T01:31:35.745511+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:38:31.345290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603937+0000","last_change":"2024-08-19T01:30:51.559622+0000","last_active":"2024-08-19T01:31:35.603937+0000","last_peered":"2024-08-19T01:31:35.603937+0000","last_clean":"2024-08-19T01:31:35.603937+0000","last_became_active":"2024-08-19T01:30:51.559263+0000","last_became_peered":"2024-08-19T01:30:51.559263+0000","last_unstale":"2024-08-19T01:31:35.603937+0000","last_undegraded":"2024-08-19T01:31:35.603937+0000","last_fullsized":"2024-08-19T01:31:35.603937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:11:25.758915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:23.672309+0000","last_change":"2024-08-19T01:30:49.540102+0000","last_active":"2024-08-19T01:32:23.672309+0000","last_peered":"2024-08-19T01:32:23.672309+0000","last_clean":"2024-08-19T01:32:23.672309+0000","last_became_active":"2024-08-19T01:30:49.539740+0000","last_became_peered":"2024-08-19T01:30:49.539740+0000","last_unstale":"2024-08-19T01:32:23.672309+0000","last_undegraded":"2024-08-19T01:32:23.672309+0000","last_fullsized":"2024-08-19T01:32:23.672309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:47:21.684925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595191+0000","last_change":"2024-08-19T01:30:58.595748+0000","last_active":"2024-08-19T01:31:35.595191+0000","last_peered":"2024-08-19T01:31:35.595191+0000","last_clean":"2024-08-19T01:31:35.595191+0000","last_became_active":"2024-08-19T01:30:58.595396+0000","last_became_peered":"2024-08-19T01:30:58.595396+0000","last_unstale":"2024-08-19T01:31:35.595191+0000","last_undegraded":"2024-08-19T01:31:35.595191+0000","last_fullsized":"2024-08-19T01:31:35.595191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:58:29.113687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604652+0000","last_change":"2024-08-19T01:30:51.559159+0000","last_active":"2024-08-19T01:31:35.604652+0000","last_peered":"2024-08-19T01:31:35.604652+0000","last_clean":"2024-08-19T01:31:35.604652+0000","last_became_active":"2024-08-19T01:30:51.558226+0000","last_became_peered":"2024-08-19T01:30:51.558226+0000","last_unstale":"2024-08-19T01:31:35.604652+0000","last_undegraded":"2024-08-19T01:31:35.604652+0000","last_fullsized":"2024-08-19T01:31:35.604652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:12:54.447301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600514+0000","last_change":"2024-08-19T01:30:49.540806+0000","last_active":"2024-08-19T01:31:35.600514+0000","last_peered":"2024-08-19T01:31:35.600514+0000","last_clean":"2024-08-19T01:31:35.600514+0000","last_became_active":"2024-08-19T01:30:49.540439+0000","last_became_peered":"2024-08-19T01:30:49.540439+0000","last_unstale":"2024-08-19T01:31:35.600514+0000","last_undegraded":"2024-08-19T01:31:35.600514+0000","last_fullsized":"2024-08-19T01:31:35.600514+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:40.088842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599613+0000","last_change":"2024-08-19T01:30:58.592661+0000","last_active":"2024-08-19T01:31:35.599613+0000","last_peered":"2024-08-19T01:31:35.599613+0000","last_clean":"2024-08-19T01:31:35.599613+0000","last_became_active":"2024-08-19T01:30:58.592066+0000","last_became_peered":"2024-08-19T01:30:58.592066+0000","last_unstale":"2024-08-19T01:31:35.599613+0000","last_undegraded":"2024-08-19T01:31:35.599613+0000","last_fullsized":"2024-08-19T01:31:35.599613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:35.039682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745099+0000","last_change":"2024-08-19T01:30:51.558497+0000","last_active":"2024-08-19T01:31:35.745099+0000","last_peered":"2024-08-19T01:31:35.745099+0000","last_clean":"2024-08-19T01:31:35.745099+0000","last_became_active":"2024-08-19T01:30:51.558185+0000","last_became_peered":"2024-08-19T01:30:51.558185+0000","last_unstale":"2024-08-19T01:31:35.745099+0000","last_undegraded":"2024-08-19T01:31:35.745099+0000","last_fullsized":"2024-08-19T01:31:35.745099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:30:42.766269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597725+0000","last_change":"2024-08-19T01:30:49.539112+0000","last_active":"2024-08-19T01:31:35.597725+0000","last_peered":"2024-08-19T01:31:35.597725+0000","last_clean":"2024-08-19T01:31:35.597725+0000","last_became_active":"2024-08-19T01:30:49.538798+0000","last_became_peered":"2024-08-19T01:30:49.538798+0000","last_unstale":"2024-08-19T01:31:35.597725+0000","last_undegraded":"2024-08-19T01:31:35.597725+0000","last_fullsized":"2024-08-19T01:31:35.597725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:10:58.977041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600778+0000","last_change":"2024-08-19T01:30:58.590727+0000","last_active":"2024-08-19T01:31:35.600778+0000","last_peered":"2024-08-19T01:31:35.600778+0000","last_clean":"2024-08-19T01:31:35.600778+0000","last_became_active":"2024-08-19T01:30:58.590208+0000","last_became_peered":"2024-08-19T01:30:58.590208+0000","last_unstale":"2024-08-19T01:31:35.600778+0000","last_undegraded":"2024-08-19T01:31:35.600778+0000","last_fullsized":"2024-08-19T01:31:35.600778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:36.422915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604177+0000","last_change":"2024-08-19T01:30:51.565113+0000","last_active":"2024-08-19T01:31:35.604177+0000","last_peered":"2024-08-19T01:31:35.604177+0000","last_clean":"2024-08-19T01:31:35.604177+0000","last_became_active":"2024-08-19T01:30:51.564769+0000","last_became_peered":"2024-08-19T01:30:51.564769+0000","last_unstale":"2024-08-19T01:31:35.604177+0000","last_undegraded":"2024-08-19T01:31:35.604177+0000","last_fullsized":"2024-08-19T01:31:35.604177+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:13:28.373114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'32","reported_seq":191,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:22.669609+0000","last_change":"2024-08-19T01:31:35.610319+0000","last_active":"2024-08-19T01:32:22.669609+0000","last_peered":"2024-08-19T01:32:22.669609+0000","last_clean":"2024-08-19T01:32:22.669609+0000","last_became_active":"2024-08-19T01:31:35.602527+0000","last_became_peered":"2024-08-19T01:31:35.602527+0000","last_unstale":"2024-08-19T01:32:22.669609+0000","last_undegraded":"2024-08-19T01:32:22.669609+0000","last_fullsized":"2024-08-19T01:32:22.669609+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:50:05.511332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62070,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":103,"num_read_kb":109,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3841,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594820+0000","last_change":"2024-08-19T01:30:58.594836+0000","last_active":"2024-08-19T01:31:35.594820+0000","last_peered":"2024-08-19T01:31:35.594820+0000","last_clean":"2024-08-19T01:31:35.594820+0000","last_became_active":"2024-08-19T01:30:58.594441+0000","last_became_peered":"2024-08-19T01:30:58.594441+0000","last_unstale":"2024-08-19T01:31:35.594820+0000","last_undegraded":"2024-08-19T01:31:35.594820+0000","last_fullsized":"2024-08-19T01:31:35.594820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:29:16.482181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596226+0000","last_change":"2024-08-19T01:30:51.562567+0000","last_active":"2024-08-19T01:31:35.596226+0000","last_peered":"2024-08-19T01:31:35.596226+0000","last_clean":"2024-08-19T01:31:35.596226+0000","last_became_active":"2024-08-19T01:30:51.562211+0000","last_became_peered":"2024-08-19T01:30:51.562211+0000","last_unstale":"2024-08-19T01:31:35.596226+0000","last_undegraded":"2024-08-19T01:31:35.596226+0000","last_fullsized":"2024-08-19T01:31:35.596226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:58.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601097+0000","last_change":"2024-08-19T01:30:49.575957+0000","last_active":"2024-08-19T01:31:35.601097+0000","last_peered":"2024-08-19T01:31:35.601097+0000","last_clean":"2024-08-19T01:31:35.601097+0000","last_became_active":"2024-08-19T01:30:49.575570+0000","last_became_peered":"2024-08-19T01:30:49.575570+0000","last_unstale":"2024-08-19T01:31:35.601097+0000","last_undegraded":"2024-08-19T01:31:35.601097+0000","last_fullsized":"2024-08-19T01:31:35.601097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:47.038113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599852+0000","last_change":"2024-08-19T01:30:58.589830+0000","last_active":"2024-08-19T01:31:35.599852+0000","last_peered":"2024-08-19T01:31:35.599852+0000","last_clean":"2024-08-19T01:31:35.599852+0000","last_became_active":"2024-08-19T01:30:58.589292+0000","last_became_peered":"2024-08-19T01:30:58.589292+0000","last_unstale":"2024-08-19T01:31:35.599852+0000","last_undegraded":"2024-08-19T01:31:35.599852+0000","last_fullsized":"2024-08-19T01:31:35.599852+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:14.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.606449+0000","last_change":"2024-08-19T01:31:35.606449+0000","last_active":"2024-08-19T01:31:35.606449+0000","last_peered":"2024-08-19T01:31:35.606449+0000","last_clean":"2024-08-19T01:31:35.606449+0000","last_became_active":"2024-08-19T01:31:35.606105+0000","last_became_peered":"2024-08-19T01:31:35.606105+0000","last_unstale":"2024-08-19T01:31:35.606449+0000","last_undegraded":"2024-08-19T01:31:35.606449+0000","last_fullsized":"2024-08-19T01:31:35.606449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:36:21.834763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605342+0000","last_change":"2024-08-19T01:30:49.542441+0000","last_active":"2024-08-19T01:31:35.605342+0000","last_peered":"2024-08-19T01:31:35.605342+0000","last_clean":"2024-08-19T01:31:35.605342+0000","last_became_active":"2024-08-19T01:30:49.542106+0000","last_became_peered":"2024-08-19T01:30:49.542106+0000","last_unstale":"2024-08-19T01:31:35.605342+0000","last_undegraded":"2024-08-19T01:31:35.605342+0000","last_fullsized":"2024-08-19T01:31:35.605342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:21:01.956282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594574+0000","last_change":"2024-08-19T01:30:58.588899+0000","last_active":"2024-08-19T01:31:35.594574+0000","last_peered":"2024-08-19T01:31:35.594574+0000","last_clean":"2024-08-19T01:31:35.594574+0000","last_became_active":"2024-08-19T01:30:58.588521+0000","last_became_peered":"2024-08-19T01:30:58.588521+0000","last_unstale":"2024-08-19T01:31:35.594574+0000","last_undegraded":"2024-08-19T01:31:35.594574+0000","last_fullsized":"2024-08-19T01:31:35.594574+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:43:59.485456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596747+0000","last_change":"2024-08-19T01:30:51.557248+0000","last_active":"2024-08-19T01:31:35.596747+0000","last_peered":"2024-08-19T01:31:35.596747+0000","last_clean":"2024-08-19T01:31:35.596747+0000","last_became_active":"2024-08-19T01:30:51.556448+0000","last_became_peered":"2024-08-19T01:30:51.556448+0000","last_unstale":"2024-08-19T01:31:35.596747+0000","last_undegraded":"2024-08-19T01:31:35.596747+0000","last_fullsized":"2024-08-19T01:31:35.596747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:19:19.587955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:21.668610+0000","last_change":"2024-08-19T01:30:49.540360+0000","last_active":"2024-08-19T01:32:21.668610+0000","last_peered":"2024-08-19T01:32:21.668610+0000","last_clean":"2024-08-19T01:32:21.668610+0000","last_became_active":"2024-08-19T01:30:49.539993+0000","last_became_peered":"2024-08-19T01:30:49.539993+0000","last_unstale":"2024-08-19T01:32:21.668610+0000","last_undegraded":"2024-08-19T01:32:21.668610+0000","last_fullsized":"2024-08-19T01:32:21.668610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:05:28.308441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":49,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868467+0000","last_change":"2024-08-19T01:30:58.592387+0000","last_active":"2024-08-19T01:31:35.868467+0000","last_peered":"2024-08-19T01:31:35.868467+0000","last_clean":"2024-08-19T01:31:35.868467+0000","last_became_active":"2024-08-19T01:30:58.592031+0000","last_became_peered":"2024-08-19T01:30:58.592031+0000","last_unstale":"2024-08-19T01:31:35.868467+0000","last_undegraded":"2024-08-19T01:31:35.868467+0000","last_fullsized":"2024-08-19T01:31:35.868467+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:31:23.159012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598112+0000","last_change":"2024-08-19T01:30:58.595650+0000","last_active":"2024-08-19T01:31:35.598112+0000","last_peered":"2024-08-19T01:31:35.598112+0000","last_clean":"2024-08-19T01:31:35.598112+0000","last_became_active":"2024-08-19T01:30:58.595304+0000","last_became_peered":"2024-08-19T01:30:58.595304+0000","last_unstale":"2024-08-19T01:31:35.598112+0000","last_undegraded":"2024-08-19T01:31:35.598112+0000","last_fullsized":"2024-08-19T01:31:35.598112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:23:53.517355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594771+0000","last_change":"2024-08-19T01:30:51.548683+0000","last_active":"2024-08-19T01:31:35.594771+0000","last_peered":"2024-08-19T01:31:35.594771+0000","last_clean":"2024-08-19T01:31:35.594771+0000","last_became_active":"2024-08-19T01:30:51.548382+0000","last_became_peered":"2024-08-19T01:30:51.548382+0000","last_unstale":"2024-08-19T01:31:35.594771+0000","last_undegraded":"2024-08-19T01:31:35.594771+0000","last_fullsized":"2024-08-19T01:31:35.594771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:46:37.551017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601362+0000","last_change":"2024-08-19T01:30:49.540136+0000","last_active":"2024-08-19T01:31:35.601362+0000","last_peered":"2024-08-19T01:31:35.601362+0000","last_clean":"2024-08-19T01:31:35.601362+0000","last_became_active":"2024-08-19T01:30:49.539680+0000","last_became_peered":"2024-08-19T01:30:49.539680+0000","last_unstale":"2024-08-19T01:31:35.601362+0000","last_undegraded":"2024-08-19T01:31:35.601362+0000","last_fullsized":"2024-08-19T01:31:35.601362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:25.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599759+0000","last_change":"2024-08-19T01:30:58.597130+0000","last_active":"2024-08-19T01:31:35.599759+0000","last_peered":"2024-08-19T01:31:35.599759+0000","last_clean":"2024-08-19T01:31:35.599759+0000","last_became_active":"2024-08-19T01:30:58.596834+0000","last_became_peered":"2024-08-19T01:30:58.596834+0000","last_unstale":"2024-08-19T01:31:35.599759+0000","last_undegraded":"2024-08-19T01:31:35.599759+0000","last_fullsized":"2024-08-19T01:31:35.599759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:13:10.441421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868410+0000","last_change":"2024-08-19T01:30:51.555575+0000","last_active":"2024-08-19T01:31:35.868410+0000","last_peered":"2024-08-19T01:31:35.868410+0000","last_clean":"2024-08-19T01:31:35.868410+0000","last_became_active":"2024-08-19T01:30:51.555309+0000","last_became_peered":"2024-08-19T01:30:51.555309+0000","last_unstale":"2024-08-19T01:31:35.868410+0000","last_undegraded":"2024-08-19T01:31:35.868410+0000","last_fullsized":"2024-08-19T01:31:35.868410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:11:25.515286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744964+0000","last_change":"2024-08-19T01:30:49.536844+0000","last_active":"2024-08-19T01:31:35.744964+0000","last_peered":"2024-08-19T01:31:35.744964+0000","last_clean":"2024-08-19T01:31:35.744964+0000","last_became_active":"2024-08-19T01:30:49.536496+0000","last_became_peered":"2024-08-19T01:30:49.536496+0000","last_unstale":"2024-08-19T01:31:35.744964+0000","last_undegraded":"2024-08-19T01:31:35.744964+0000","last_fullsized":"2024-08-19T01:31:35.744964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:04.917188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598260+0000","last_change":"2024-08-19T01:30:49.539356+0000","last_active":"2024-08-19T01:31:35.598260+0000","last_peered":"2024-08-19T01:31:35.598260+0000","last_clean":"2024-08-19T01:31:35.598260+0000","last_became_active":"2024-08-19T01:30:49.538953+0000","last_became_peered":"2024-08-19T01:30:49.538953+0000","last_unstale":"2024-08-19T01:31:35.598260+0000","last_undegraded":"2024-08-19T01:31:35.598260+0000","last_fullsized":"2024-08-19T01:31:35.598260+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:13:17.845140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.870021+0000","last_change":"2024-08-19T01:31:35.870021+0000","last_active":"2024-08-19T01:31:35.870021+0000","last_peered":"2024-08-19T01:31:35.870021+0000","last_clean":"2024-08-19T01:31:35.870021+0000","last_became_active":"2024-08-19T01:31:35.869742+0000","last_became_peered":"2024-08-19T01:31:35.869742+0000","last_unstale":"2024-08-19T01:31:35.870021+0000","last_undegraded":"2024-08-19T01:31:35.870021+0000","last_fullsized":"2024-08-19T01:31:35.870021+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:51.848694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597559+0000","last_change":"2024-08-19T01:30:58.590385+0000","last_active":"2024-08-19T01:31:35.597559+0000","last_peered":"2024-08-19T01:31:35.597559+0000","last_clean":"2024-08-19T01:31:35.597559+0000","last_became_active":"2024-08-19T01:30:58.590004+0000","last_became_peered":"2024-08-19T01:30:58.590004+0000","last_unstale":"2024-08-19T01:31:35.597559+0000","last_undegraded":"2024-08-19T01:31:35.597559+0000","last_fullsized":"2024-08-19T01:31:35.597559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:13:01.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:21.669153+0000","last_change":"2024-08-19T01:31:35.614962+0000","last_active":"2024-08-19T01:32:21.669153+0000","last_peered":"2024-08-19T01:32:21.669153+0000","last_clean":"2024-08-19T01:32:21.669153+0000","last_became_active":"2024-08-19T01:31:35.605904+0000","last_became_peered":"2024-08-19T01:31:35.605904+0000","last_unstale":"2024-08-19T01:32:21.669153+0000","last_undegraded":"2024-08-19T01:32:21.669153+0000","last_fullsized":"2024-08-19T01:32:21.669153+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:26:36.102805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603484+0000","last_change":"2024-08-19T01:30:51.550448+0000","last_active":"2024-08-19T01:31:35.603484+0000","last_peered":"2024-08-19T01:31:35.603484+0000","last_clean":"2024-08-19T01:31:35.603484+0000","last_became_active":"2024-08-19T01:30:51.550074+0000","last_became_peered":"2024-08-19T01:30:51.550074+0000","last_unstale":"2024-08-19T01:31:35.603484+0000","last_undegraded":"2024-08-19T01:31:35.603484+0000","last_fullsized":"2024-08-19T01:31:35.603484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:33:13.016183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597794+0000","last_change":"2024-08-19T01:30:58.592077+0000","last_active":"2024-08-19T01:31:35.597794+0000","last_peered":"2024-08-19T01:31:35.597794+0000","last_clean":"2024-08-19T01:31:35.597794+0000","last_became_active":"2024-08-19T01:30:58.591651+0000","last_became_peered":"2024-08-19T01:30:58.591651+0000","last_unstale":"2024-08-19T01:31:35.597794+0000","last_undegraded":"2024-08-19T01:31:35.597794+0000","last_fullsized":"2024-08-19T01:31:35.597794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:09:24.980954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599538+0000","last_change":"2024-08-19T01:30:51.552772+0000","last_active":"2024-08-19T01:31:35.599538+0000","last_peered":"2024-08-19T01:31:35.599538+0000","last_clean":"2024-08-19T01:31:35.599538+0000","last_became_active":"2024-08-19T01:30:51.550904+0000","last_became_peered":"2024-08-19T01:30:51.550904+0000","last_unstale":"2024-08-19T01:31:35.599538+0000","last_undegraded":"2024-08-19T01:31:35.599538+0000","last_fullsized":"2024-08-19T01:31:35.599538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:21:50.082587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598023+0000","last_change":"2024-08-19T01:30:51.559756+0000","last_active":"2024-08-19T01:31:35.598023+0000","last_peered":"2024-08-19T01:31:35.598023+0000","last_clean":"2024-08-19T01:31:35.598023+0000","last_became_active":"2024-08-19T01:30:51.559370+0000","last_became_peered":"2024-08-19T01:30:51.559370+0000","last_unstale":"2024-08-19T01:31:35.598023+0000","last_undegraded":"2024-08-19T01:31:35.598023+0000","last_fullsized":"2024-08-19T01:31:35.598023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:27:30.525346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597386+0000","last_change":"2024-08-19T01:30:51.556577+0000","last_active":"2024-08-19T01:31:35.597386+0000","last_peered":"2024-08-19T01:31:35.597386+0000","last_clean":"2024-08-19T01:31:35.597386+0000","last_became_active":"2024-08-19T01:30:51.555929+0000","last_became_peered":"2024-08-19T01:30:51.555929+0000","last_unstale":"2024-08-19T01:31:35.597386+0000","last_undegraded":"2024-08-19T01:31:35.597386+0000","last_fullsized":"2024-08-19T01:31:35.597386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:46:31.660752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598037+0000","last_change":"2024-08-19T01:30:51.558626+0000","last_active":"2024-08-19T01:31:35.598037+0000","last_peered":"2024-08-19T01:31:35.598037+0000","last_clean":"2024-08-19T01:31:35.598037+0000","last_became_active":"2024-08-19T01:30:51.558188+0000","last_became_peered":"2024-08-19T01:30:51.558188+0000","last_unstale":"2024-08-19T01:31:35.598037+0000","last_undegraded":"2024-08-19T01:31:35.598037+0000","last_fullsized":"2024-08-19T01:31:35.598037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:43:44.777643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745491+0000","last_change":"2024-08-19T01:30:51.552870+0000","last_active":"2024-08-19T01:31:35.745491+0000","last_peered":"2024-08-19T01:31:35.745491+0000","last_clean":"2024-08-19T01:31:35.745491+0000","last_became_active":"2024-08-19T01:30:51.552514+0000","last_became_peered":"2024-08-19T01:30:51.552514+0000","last_unstale":"2024-08-19T01:31:35.745491+0000","last_undegraded":"2024-08-19T01:31:35.745491+0000","last_fullsized":"2024-08-19T01:31:35.745491+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:10:58.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600636+0000","last_change":"2024-08-19T01:30:51.554216+0000","last_active":"2024-08-19T01:31:35.600636+0000","last_peered":"2024-08-19T01:31:35.600636+0000","last_clean":"2024-08-19T01:31:35.600636+0000","last_became_active":"2024-08-19T01:30:51.552535+0000","last_became_peered":"2024-08-19T01:30:51.552535+0000","last_unstale":"2024-08-19T01:31:35.600636+0000","last_undegraded":"2024-08-19T01:31:35.600636+0000","last_fullsized":"2024-08-19T01:31:35.600636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:26:52.971897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598164+0000","last_change":"2024-08-19T01:30:51.561703+0000","last_active":"2024-08-19T01:31:35.598164+0000","last_peered":"2024-08-19T01:31:35.598164+0000","last_clean":"2024-08-19T01:31:35.598164+0000","last_became_active":"2024-08-19T01:30:51.561385+0000","last_became_peered":"2024-08-19T01:30:51.561385+0000","last_unstale":"2024-08-19T01:31:35.598164+0000","last_undegraded":"2024-08-19T01:31:35.598164+0000","last_fullsized":"2024-08-19T01:31:35.598164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:07:23.187023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868940+0000","last_change":"2024-08-19T01:30:51.564047+0000","last_active":"2024-08-19T01:31:35.868940+0000","last_peered":"2024-08-19T01:31:35.868940+0000","last_clean":"2024-08-19T01:31:35.868940+0000","last_became_active":"2024-08-19T01:30:51.563743+0000","last_became_peered":"2024-08-19T01:30:51.563743+0000","last_unstale":"2024-08-19T01:31:35.868940+0000","last_undegraded":"2024-08-19T01:31:35.868940+0000","last_fullsized":"2024-08-19T01:31:35.868940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:50:37.639496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601317+0000","last_change":"2024-08-19T01:30:51.557058+0000","last_active":"2024-08-19T01:31:35.601317+0000","last_peered":"2024-08-19T01:31:35.601317+0000","last_clean":"2024-08-19T01:31:35.601317+0000","last_became_active":"2024-08-19T01:30:51.554611+0000","last_became_peered":"2024-08-19T01:30:51.554611+0000","last_unstale":"2024-08-19T01:31:35.601317+0000","last_undegraded":"2024-08-19T01:31:35.601317+0000","last_fullsized":"2024-08-19T01:31:35.601317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:56:25.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868614+0000","last_change":"2024-08-19T01:30:51.564119+0000","last_active":"2024-08-19T01:31:35.868614+0000","last_peered":"2024-08-19T01:31:35.868614+0000","last_clean":"2024-08-19T01:31:35.868614+0000","last_became_active":"2024-08-19T01:30:51.563840+0000","last_became_peered":"2024-08-19T01:30:51.563840+0000","last_unstale":"2024-08-19T01:31:35.868614+0000","last_undegraded":"2024-08-19T01:31:35.868614+0000","last_fullsized":"2024-08-19T01:31:35.868614+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:44:44.928760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596119+0000","last_change":"2024-08-19T01:30:51.561639+0000","last_active":"2024-08-19T01:31:35.596119+0000","last_peered":"2024-08-19T01:31:35.596119+0000","last_clean":"2024-08-19T01:31:35.596119+0000","last_became_active":"2024-08-19T01:30:51.561207+0000","last_became_peered":"2024-08-19T01:30:51.561207+0000","last_unstale":"2024-08-19T01:31:35.596119+0000","last_undegraded":"2024-08-19T01:31:35.596119+0000","last_fullsized":"2024-08-19T01:31:35.596119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:12:20.076628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745343+0000","last_change":"2024-08-19T01:30:51.556700+0000","last_active":"2024-08-19T01:31:35.745343+0000","last_peered":"2024-08-19T01:31:35.745343+0000","last_clean":"2024-08-19T01:31:35.745343+0000","last_became_active":"2024-08-19T01:30:51.556407+0000","last_became_peered":"2024-08-19T01:30:51.556407+0000","last_unstale":"2024-08-19T01:31:35.745343+0000","last_undegraded":"2024-08-19T01:31:35.745343+0000","last_fullsized":"2024-08-19T01:31:35.745343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:03:16.664253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599693+0000","last_change":"2024-08-19T01:30:51.553764+0000","last_active":"2024-08-19T01:31:35.599693+0000","last_peered":"2024-08-19T01:31:35.599693+0000","last_clean":"2024-08-19T01:31:35.599693+0000","last_became_active":"2024-08-19T01:30:51.553442+0000","last_became_peered":"2024-08-19T01:30:51.553442+0000","last_unstale":"2024-08-19T01:31:35.599693+0000","last_undegraded":"2024-08-19T01:31:35.599693+0000","last_fullsized":"2024-08-19T01:31:35.599693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:36:44.476979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598004+0000","last_change":"2024-08-19T01:30:51.565528+0000","last_active":"2024-08-19T01:31:35.598004+0000","last_peered":"2024-08-19T01:31:35.598004+0000","last_clean":"2024-08-19T01:31:35.598004+0000","last_became_active":"2024-08-19T01:30:51.564649+0000","last_became_peered":"2024-08-19T01:30:51.564649+0000","last_unstale":"2024-08-19T01:31:35.598004+0000","last_undegraded":"2024-08-19T01:31:35.598004+0000","last_fullsized":"2024-08-19T01:31:35.598004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:04:12.868496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595505+0000","last_change":"2024-08-19T01:30:51.553425+0000","last_active":"2024-08-19T01:31:35.595505+0000","last_peered":"2024-08-19T01:31:35.595505+0000","last_clean":"2024-08-19T01:31:35.595505+0000","last_became_active":"2024-08-19T01:30:51.553123+0000","last_became_peered":"2024-08-19T01:30:51.553123+0000","last_unstale":"2024-08-19T01:31:35.595505+0000","last_undegraded":"2024-08-19T01:31:35.595505+0000","last_fullsized":"2024-08-19T01:31:35.595505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:07.969228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600659+0000","last_change":"2024-08-19T01:30:51.556820+0000","last_active":"2024-08-19T01:31:35.600659+0000","last_peered":"2024-08-19T01:31:35.600659+0000","last_clean":"2024-08-19T01:31:35.600659+0000","last_became_active":"2024-08-19T01:30:51.556532+0000","last_became_peered":"2024-08-19T01:30:51.556532+0000","last_unstale":"2024-08-19T01:31:35.600659+0000","last_undegraded":"2024-08-19T01:31:35.600659+0000","last_fullsized":"2024-08-19T01:31:35.600659+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:40:37.053193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595799+0000","last_change":"2024-08-19T01:30:51.561610+0000","last_active":"2024-08-19T01:31:35.595799+0000","last_peered":"2024-08-19T01:31:35.595799+0000","last_clean":"2024-08-19T01:31:35.595799+0000","last_became_active":"2024-08-19T01:30:51.561270+0000","last_became_peered":"2024-08-19T01:30:51.561270+0000","last_unstale":"2024-08-19T01:31:35.595799+0000","last_undegraded":"2024-08-19T01:31:35.595799+0000","last_fullsized":"2024-08-19T01:31:35.595799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:15:15.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597839+0000","last_change":"2024-08-19T01:30:51.558128+0000","last_active":"2024-08-19T01:31:35.597839+0000","last_peered":"2024-08-19T01:31:35.597839+0000","last_clean":"2024-08-19T01:31:35.597839+0000","last_became_active":"2024-08-19T01:30:51.557785+0000","last_became_peered":"2024-08-19T01:30:51.557785+0000","last_unstale":"2024-08-19T01:31:35.597839+0000","last_undegraded":"2024-08-19T01:31:35.597839+0000","last_fullsized":"2024-08-19T01:31:35.597839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:14.849212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597900+0000","last_change":"2024-08-19T01:30:51.553917+0000","last_active":"2024-08-19T01:31:35.597900+0000","last_peered":"2024-08-19T01:31:35.597900+0000","last_clean":"2024-08-19T01:31:35.597900+0000","last_became_active":"2024-08-19T01:30:51.553596+0000","last_became_peered":"2024-08-19T01:30:51.553596+0000","last_unstale":"2024-08-19T01:31:35.597900+0000","last_undegraded":"2024-08-19T01:31:35.597900+0000","last_fullsized":"2024-08-19T01:31:35.597900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:03:36.001486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745336+0000","last_change":"2024-08-19T01:30:51.555201+0000","last_active":"2024-08-19T01:31:35.745336+0000","last_peered":"2024-08-19T01:31:35.745336+0000","last_clean":"2024-08-19T01:31:35.745336+0000","last_became_active":"2024-08-19T01:30:51.554160+0000","last_became_peered":"2024-08-19T01:30:51.554160+0000","last_unstale":"2024-08-19T01:31:35.745336+0000","last_undegraded":"2024-08-19T01:31:35.745336+0000","last_fullsized":"2024-08-19T01:31:35.745336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:58:20.586931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600619+0000","last_change":"2024-08-19T01:30:51.560517+0000","last_active":"2024-08-19T01:31:35.600619+0000","last_peered":"2024-08-19T01:31:35.600619+0000","last_clean":"2024-08-19T01:31:35.600619+0000","last_became_active":"2024-08-19T01:30:51.557454+0000","last_became_peered":"2024-08-19T01:30:51.557454+0000","last_unstale":"2024-08-19T01:31:35.600619+0000","last_undegraded":"2024-08-19T01:31:35.600619+0000","last_fullsized":"2024-08-19T01:31:35.600619+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:41:08.192165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:51.565961+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:51.565491+0000","last_became_peered":"2024-08-19T01:30:51.565491+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:53:06.263411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599327+0000","last_change":"2024-08-19T01:30:51.551297+0000","last_active":"2024-08-19T01:31:35.599327+0000","last_peered":"2024-08-19T01:31:35.599327+0000","last_clean":"2024-08-19T01:31:35.599327+0000","last_became_active":"2024-08-19T01:30:51.550929+0000","last_became_peered":"2024-08-19T01:30:51.550929+0000","last_unstale":"2024-08-19T01:31:35.599327+0000","last_undegraded":"2024-08-19T01:31:35.599327+0000","last_fullsized":"2024-08-19T01:31:35.599327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:07:03.625360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603232+0000","last_change":"2024-08-19T01:31:35.602883+0000","last_active":"2024-08-19T01:31:35.603232+0000","last_peered":"2024-08-19T01:31:35.603232+0000","last_clean":"2024-08-19T01:31:35.603232+0000","last_became_active":"2024-08-19T01:31:35.602571+0000","last_became_peered":"2024-08-19T01:31:35.602571+0000","last_unstale":"2024-08-19T01:31:35.603232+0000","last_undegraded":"2024-08-19T01:31:35.603232+0000","last_fullsized":"2024-08-19T01:31:35.603232+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:08:13.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868405+0000","last_change":"2024-08-19T01:30:51.552548+0000","last_active":"2024-08-19T01:31:35.868405+0000","last_peered":"2024-08-19T01:31:35.868405+0000","last_clean":"2024-08-19T01:31:35.868405+0000","last_became_active":"2024-08-19T01:30:51.552264+0000","last_became_peered":"2024-08-19T01:30:51.552264+0000","last_unstale":"2024-08-19T01:31:35.868405+0000","last_undegraded":"2024-08-19T01:31:35.868405+0000","last_fullsized":"2024-08-19T01:31:35.868405+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:17:53.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603728+0000","last_change":"2024-08-19T01:30:51.558686+0000","last_active":"2024-08-19T01:31:35.603728+0000","last_peered":"2024-08-19T01:31:35.603728+0000","last_clean":"2024-08-19T01:31:35.603728+0000","last_became_active":"2024-08-19T01:30:51.558190+0000","last_became_peered":"2024-08-19T01:30:51.558190+0000","last_unstale":"2024-08-19T01:31:35.603728+0000","last_undegraded":"2024-08-19T01:31:35.603728+0000","last_fullsized":"2024-08-19T01:31:35.603728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:52:33.056948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598221+0000","last_change":"2024-08-19T01:30:51.548868+0000","last_active":"2024-08-19T01:31:35.598221+0000","last_peered":"2024-08-19T01:31:35.598221+0000","last_clean":"2024-08-19T01:31:35.598221+0000","last_became_active":"2024-08-19T01:30:51.548347+0000","last_became_peered":"2024-08-19T01:30:51.548347+0000","last_unstale":"2024-08-19T01:31:35.598221+0000","last_undegraded":"2024-08-19T01:31:35.598221+0000","last_fullsized":"2024-08-19T01:31:35.598221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:39:22.049290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603369+0000","last_change":"2024-08-19T01:30:51.554054+0000","last_active":"2024-08-19T01:31:35.603369+0000","last_peered":"2024-08-19T01:31:35.603369+0000","last_clean":"2024-08-19T01:31:35.603369+0000","last_became_active":"2024-08-19T01:30:51.553731+0000","last_became_peered":"2024-08-19T01:30:51.553731+0000","last_unstale":"2024-08-19T01:31:35.603369+0000","last_undegraded":"2024-08-19T01:31:35.603369+0000","last_fullsized":"2024-08-19T01:31:35.603369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:25:17.939024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600261+0000","last_change":"2024-08-19T01:30:51.558662+0000","last_active":"2024-08-19T01:31:35.600261+0000","last_peered":"2024-08-19T01:31:35.600261+0000","last_clean":"2024-08-19T01:31:35.600261+0000","last_became_active":"2024-08-19T01:30:51.558131+0000","last_became_peered":"2024-08-19T01:30:51.558131+0000","last_unstale":"2024-08-19T01:31:35.600261+0000","last_undegraded":"2024-08-19T01:31:35.600261+0000","last_fullsized":"2024-08-19T01:31:35.600261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:31:25.054005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598117+0000","last_change":"2024-08-19T01:30:51.565170+0000","last_active":"2024-08-19T01:31:35.598117+0000","last_peered":"2024-08-19T01:31:35.598117+0000","last_clean":"2024-08-19T01:31:35.598117+0000","last_became_active":"2024-08-19T01:30:51.564765+0000","last_became_peered":"2024-08-19T01:30:51.564765+0000","last_unstale":"2024-08-19T01:31:35.598117+0000","last_undegraded":"2024-08-19T01:31:35.598117+0000","last_fullsized":"2024-08-19T01:31:35.598117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:00:48.668865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598106+0000","last_change":"2024-08-19T01:30:51.555300+0000","last_active":"2024-08-19T01:31:35.598106+0000","last_peered":"2024-08-19T01:31:35.598106+0000","last_clean":"2024-08-19T01:31:35.598106+0000","last_became_active":"2024-08-19T01:30:51.554977+0000","last_became_peered":"2024-08-19T01:30:51.554977+0000","last_unstale":"2024-08-19T01:31:35.598106+0000","last_undegraded":"2024-08-19T01:31:35.598106+0000","last_fullsized":"2024-08-19T01:31:35.598106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:18:58.961783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600410+0000","last_change":"2024-08-19T01:30:51.557906+0000","last_active":"2024-08-19T01:31:35.600410+0000","last_peered":"2024-08-19T01:31:35.600410+0000","last_clean":"2024-08-19T01:31:35.600410+0000","last_became_active":"2024-08-19T01:30:51.557598+0000","last_became_peered":"2024-08-19T01:30:51.557598+0000","last_unstale":"2024-08-19T01:31:35.600410+0000","last_undegraded":"2024-08-19T01:31:35.600410+0000","last_fullsized":"2024-08-19T01:31:35.600410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:33:12.785910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66460,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":320,"num_read_kb":226,"num_write":168,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4463,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":199380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282761,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481824,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":513466,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743460,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32296,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712856,"statfs":{"total":95995035648,"available":95961964544,"internally_reserved":0,"allocated":1146880,"data_stored":1163194,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906986,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":517798,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168621,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":513622,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430258,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1109920,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593784,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":508878,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822716,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1105118,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986241,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":575988,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215172,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":509072,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":509516,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":571870,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":510266,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T01:32:29.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T01:32:29.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:29 smithi123 ceph-mon[21829]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:29.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:29 smithi161 ceph-mon[21757]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:30.121 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:30.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 -- 172.21.15.64:0/2170697682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 msgr2=0x7fa5641078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:30.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/2170697682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641078b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa550009930 tx=0x7fa55002a420 comp rx=0 tx=0).stop 2024-08-19T01:32:30.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 -- 172.21.15.64:0/2170697682 shutdown_connections 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/2170697682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564107fc0 0x7fa56410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/2170697682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641078b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/2170697682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa564102ac0 0x7fa564102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 -- 172.21.15.64:0/2170697682 >> 172.21.15.64:0/2170697682 conn(0x7fa5640fc500 msgr2=0x7fa5640fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 -- 172.21.15.64:0/2170697682 shutdown_connections 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 -- 172.21.15.64:0/2170697682 wait complete. 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 Processor -- start 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.645+0000 7fa56aaf2640 1 -- start start 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56aaf2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa564102ac0 0x7fa564198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56aaf2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56aaf2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564107fc0 0x7fa56419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56aaf2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa564111e60 con 0x7fa564103470 2024-08-19T01:32:30.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56aaf2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa564111ce0 con 0x7fa564102ac0 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56aaf2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa564111fe0 con 0x7fa564107fc0 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45458/0 (socket says 172.21.15.64:45458) 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 -- 172.21.15.64:0/180309686 learned_addr learned my addr 172.21.15.64:0/180309686 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56888e640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa564102ac0 0x7fa564198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa56908f640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564107fc0 0x7fa56419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 -- 172.21.15.64:0/180309686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564107fc0 msgr2=0x7fa56419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564107fc0 0x7fa56419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 -- 172.21.15.64:0/180309686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa564102ac0 msgr2=0x7fa564198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa564102ac0 0x7fa564198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 -- 172.21.15.64:0/180309686 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56419d410 con 0x7fa564103470 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa563fff640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641989a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa550009230 tx=0x7fa5500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:30.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.646+0000 7fa561ffb640 1 -- 172.21.15.64:0/180309686 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55003f030 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.647+0000 7fa561ffb640 1 -- 172.21.15.64:0/180309686 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa550003ce0 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.647+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56419d6a0 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.647+0000 7fa561ffb640 1 -- 172.21.15.64:0/180309686 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550035070 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.647+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5641a4fe0 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.648+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa56406a520 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.649+0000 7fa561ffb640 1 -- 172.21.15.64:0/180309686 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa550042030 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.649+0000 7fa561ffb640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5540a2c10 0x7fa5540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.650+0000 7fa56888e640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5540a2c10 0x7fa5540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.650+0000 7fa561ffb640 1 -- 172.21.15.64:0/180309686 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa5500efba0 con 0x7fa564103470 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.650+0000 7fa56888e640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5540a2c10 0x7fa5540a50d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa558009890 tx=0x7fa558008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:30.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.653+0000 7fa561ffb640 1 -- 172.21.15.64:0/180309686 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa5500bc640 con 0x7fa564103470 2024-08-19T01:32:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.825+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa564062360 con 0x7fa5540a2c10 2024-08-19T01:32:30.826 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:30 smithi064 ceph-mon[18515]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:32:30.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.834+0000 7fa561ffb640 1 -- 172.21.15.64:0/180309686 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340636 (secure 0 0 0) 0x7fa564062360 con 0x7fa5540a2c10 2024-08-19T01:32:30.838 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:30.840 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5540a2c10 msgr2=0x7fa5540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5540a2c10 0x7fa5540a50d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa558009890 tx=0x7fa558008040 comp rx=0 tx=0).stop 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 msgr2=0x7fa5641989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641989a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa550009230 tx=0x7fa5500042f0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 shutdown_connections 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa5540a2c10 0x7fa5540a50d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564107fc0 0x7fa56419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa564103470 0x7fa5641989a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 --2- 172.21.15.64:0/180309686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa564102ac0 0x7fa564198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 >> 172.21.15.64:0/180309686 conn(0x7fa5640fc500 msgr2=0x7fa564105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:30.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 shutdown_connections 2024-08-19T01:32:30.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:30.837+0000 7fa56aaf2640 1 -- 172.21.15.64:0/180309686 wait complete. 2024-08-19T01:32:30.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:30 smithi123 ceph-mon[21829]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:32:30.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:30 smithi161 ceph-mon[21757]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:32:31.550 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-08-19T01:32:29.697811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657062,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":476,"num_read_kb":355,"num_write":351,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1185199,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361424,"kb_used_data":12248,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580400,"statfs":{"total":1151940427776,"available":1151570329600,"internally_reserved":0,"allocated":12541952,"data_stored":8108708,"data_compressed":390537,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003223"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595389+0000","last_change":"2024-08-19T01:30:49.539512+0000","last_active":"2024-08-19T01:31:35.595389+0000","last_peered":"2024-08-19T01:31:35.595389+0000","last_clean":"2024-08-19T01:31:35.595389+0000","last_became_active":"2024-08-19T01:30:49.539163+0000","last_became_peered":"2024-08-19T01:30:49.539163+0000","last_unstale":"2024-08-19T01:31:35.595389+0000","last_undegraded":"2024-08-19T01:31:35.595389+0000","last_fullsized":"2024-08-19T01:31:35.595389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:23.874916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868682+0000","last_change":"2024-08-19T01:30:51.561135+0000","last_active":"2024-08-19T01:31:35.868682+0000","last_peered":"2024-08-19T01:31:35.868682+0000","last_clean":"2024-08-19T01:31:35.868682+0000","last_became_active":"2024-08-19T01:30:51.560822+0000","last_became_peered":"2024-08-19T01:30:51.560822+0000","last_unstale":"2024-08-19T01:31:35.868682+0000","last_undegraded":"2024-08-19T01:31:35.868682+0000","last_fullsized":"2024-08-19T01:31:35.868682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:08:35.918895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597319+0000","last_change":"2024-08-19T01:30:58.595727+0000","last_active":"2024-08-19T01:31:35.597319+0000","last_peered":"2024-08-19T01:31:35.597319+0000","last_clean":"2024-08-19T01:31:35.597319+0000","last_became_active":"2024-08-19T01:30:58.595286+0000","last_became_peered":"2024-08-19T01:30:58.595286+0000","last_unstale":"2024-08-19T01:31:35.597319+0000","last_undegraded":"2024-08-19T01:31:35.597319+0000","last_fullsized":"2024-08-19T01:31:35.597319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:25:02.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868994+0000","last_change":"2024-08-19T01:30:49.575693+0000","last_active":"2024-08-19T01:31:35.868994+0000","last_peered":"2024-08-19T01:31:35.868994+0000","last_clean":"2024-08-19T01:31:35.868994+0000","last_became_active":"2024-08-19T01:30:49.575331+0000","last_became_peered":"2024-08-19T01:30:49.575331+0000","last_unstale":"2024-08-19T01:31:35.868994+0000","last_undegraded":"2024-08-19T01:31:35.868994+0000","last_fullsized":"2024-08-19T01:31:35.868994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:31:50.773160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598318+0000","last_change":"2024-08-19T01:30:51.562446+0000","last_active":"2024-08-19T01:31:35.598318+0000","last_peered":"2024-08-19T01:31:35.598318+0000","last_clean":"2024-08-19T01:31:35.598318+0000","last_became_active":"2024-08-19T01:30:51.561475+0000","last_became_peered":"2024-08-19T01:30:51.561475+0000","last_unstale":"2024-08-19T01:31:35.598318+0000","last_undegraded":"2024-08-19T01:31:35.598318+0000","last_fullsized":"2024-08-19T01:31:35.598318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:51:41.531247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599871+0000","last_change":"2024-08-19T01:30:58.591371+0000","last_active":"2024-08-19T01:31:35.599871+0000","last_peered":"2024-08-19T01:31:35.599871+0000","last_clean":"2024-08-19T01:31:35.599871+0000","last_became_active":"2024-08-19T01:30:58.590936+0000","last_became_peered":"2024-08-19T01:30:58.590936+0000","last_unstale":"2024-08-19T01:31:35.599871+0000","last_undegraded":"2024-08-19T01:31:35.599871+0000","last_fullsized":"2024-08-19T01:31:35.599871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:07:33.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.613588+0000","last_change":"2024-08-19T01:31:35.613145+0000","last_active":"2024-08-19T01:31:35.613588+0000","last_peered":"2024-08-19T01:31:35.613588+0000","last_clean":"2024-08-19T01:31:35.613588+0000","last_became_active":"2024-08-19T01:31:35.605087+0000","last_became_peered":"2024-08-19T01:31:35.605087+0000","last_unstale":"2024-08-19T01:31:35.613588+0000","last_undegraded":"2024-08-19T01:31:35.613588+0000","last_fullsized":"2024-08-19T01:31:35.613588+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:25:59.109707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596225+0000","last_change":"2024-08-19T01:30:51.555256+0000","last_active":"2024-08-19T01:31:35.596225+0000","last_peered":"2024-08-19T01:31:35.596225+0000","last_clean":"2024-08-19T01:31:35.596225+0000","last_became_active":"2024-08-19T01:30:51.554030+0000","last_became_peered":"2024-08-19T01:30:51.554030+0000","last_unstale":"2024-08-19T01:31:35.596225+0000","last_undegraded":"2024-08-19T01:31:35.596225+0000","last_fullsized":"2024-08-19T01:31:35.596225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:12:04.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600498+0000","last_change":"2024-08-19T01:30:58.589362+0000","last_active":"2024-08-19T01:31:35.600498+0000","last_peered":"2024-08-19T01:31:35.600498+0000","last_clean":"2024-08-19T01:31:35.600498+0000","last_became_active":"2024-08-19T01:30:58.588937+0000","last_became_peered":"2024-08-19T01:30:58.588937+0000","last_unstale":"2024-08-19T01:31:35.600498+0000","last_undegraded":"2024-08-19T01:31:35.600498+0000","last_fullsized":"2024-08-19T01:31:35.600498+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:31.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604267+0000","last_change":"2024-08-19T01:30:49.538597+0000","last_active":"2024-08-19T01:31:35.604267+0000","last_peered":"2024-08-19T01:31:35.604267+0000","last_clean":"2024-08-19T01:31:35.604267+0000","last_became_active":"2024-08-19T01:30:49.538228+0000","last_became_peered":"2024-08-19T01:30:49.538228+0000","last_unstale":"2024-08-19T01:31:35.604267+0000","last_undegraded":"2024-08-19T01:31:35.604267+0000","last_fullsized":"2024-08-19T01:31:35.604267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:31.350962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600891+0000","last_change":"2024-08-19T01:30:51.562829+0000","last_active":"2024-08-19T01:31:35.600891+0000","last_peered":"2024-08-19T01:31:35.600891+0000","last_clean":"2024-08-19T01:31:35.600891+0000","last_became_active":"2024-08-19T01:30:51.562144+0000","last_became_peered":"2024-08-19T01:30:51.562144+0000","last_unstale":"2024-08-19T01:31:35.600891+0000","last_undegraded":"2024-08-19T01:31:35.600891+0000","last_fullsized":"2024-08-19T01:31:35.600891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:17:09.184222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599645+0000","last_change":"2024-08-19T01:30:58.593178+0000","last_active":"2024-08-19T01:31:35.599645+0000","last_peered":"2024-08-19T01:31:35.599645+0000","last_clean":"2024-08-19T01:31:35.599645+0000","last_became_active":"2024-08-19T01:30:58.591247+0000","last_became_peered":"2024-08-19T01:30:58.591247+0000","last_unstale":"2024-08-19T01:31:35.599645+0000","last_undegraded":"2024-08-19T01:31:35.599645+0000","last_fullsized":"2024-08-19T01:31:35.599645+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:20:40.796135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604428+0000","last_change":"2024-08-19T01:30:51.562307+0000","last_active":"2024-08-19T01:31:35.604428+0000","last_peered":"2024-08-19T01:31:35.604428+0000","last_clean":"2024-08-19T01:31:35.604428+0000","last_became_active":"2024-08-19T01:30:51.561957+0000","last_became_peered":"2024-08-19T01:30:51.561957+0000","last_unstale":"2024-08-19T01:31:35.604428+0000","last_undegraded":"2024-08-19T01:31:35.604428+0000","last_fullsized":"2024-08-19T01:31:35.604428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:22.902317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.610093+0000","last_change":"2024-08-19T01:31:35.609612+0000","last_active":"2024-08-19T01:31:35.610093+0000","last_peered":"2024-08-19T01:31:35.610093+0000","last_clean":"2024-08-19T01:31:35.610093+0000","last_became_active":"2024-08-19T01:31:35.604969+0000","last_became_peered":"2024-08-19T01:31:35.604969+0000","last_unstale":"2024-08-19T01:31:35.610093+0000","last_undegraded":"2024-08-19T01:31:35.610093+0000","last_fullsized":"2024-08-19T01:31:35.610093+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:59:38.089119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600108+0000","last_change":"2024-08-19T01:30:58.588984+0000","last_active":"2024-08-19T01:31:35.600108+0000","last_peered":"2024-08-19T01:31:35.600108+0000","last_clean":"2024-08-19T01:31:35.600108+0000","last_became_active":"2024-08-19T01:30:58.588504+0000","last_became_peered":"2024-08-19T01:30:58.588504+0000","last_unstale":"2024-08-19T01:31:35.600108+0000","last_undegraded":"2024-08-19T01:31:35.600108+0000","last_fullsized":"2024-08-19T01:31:35.600108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:29:14.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869072+0000","last_change":"2024-08-19T01:30:51.553293+0000","last_active":"2024-08-19T01:31:35.869072+0000","last_peered":"2024-08-19T01:31:35.869072+0000","last_clean":"2024-08-19T01:31:35.869072+0000","last_became_active":"2024-08-19T01:30:51.552990+0000","last_became_peered":"2024-08-19T01:30:51.552990+0000","last_unstale":"2024-08-19T01:31:35.869072+0000","last_undegraded":"2024-08-19T01:31:35.869072+0000","last_fullsized":"2024-08-19T01:31:35.869072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:28:33.193014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597523+0000","last_change":"2024-08-19T01:30:49.537742+0000","last_active":"2024-08-19T01:31:35.597523+0000","last_peered":"2024-08-19T01:31:35.597523+0000","last_clean":"2024-08-19T01:31:35.597523+0000","last_became_active":"2024-08-19T01:30:49.537153+0000","last_became_peered":"2024-08-19T01:30:49.537153+0000","last_unstale":"2024-08-19T01:31:35.597523+0000","last_undegraded":"2024-08-19T01:31:35.597523+0000","last_fullsized":"2024-08-19T01:31:35.597523+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:25:03.650900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600493+0000","last_change":"2024-08-19T01:30:58.592348+0000","last_active":"2024-08-19T01:31:35.600493+0000","last_peered":"2024-08-19T01:31:35.600493+0000","last_clean":"2024-08-19T01:31:35.600493+0000","last_became_active":"2024-08-19T01:30:58.591641+0000","last_became_peered":"2024-08-19T01:30:58.591641+0000","last_unstale":"2024-08-19T01:31:35.600493+0000","last_undegraded":"2024-08-19T01:31:35.600493+0000","last_fullsized":"2024-08-19T01:31:35.600493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:39:17.542800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599683+0000","last_change":"2024-08-19T01:30:51.552126+0000","last_active":"2024-08-19T01:31:35.599683+0000","last_peered":"2024-08-19T01:31:35.599683+0000","last_clean":"2024-08-19T01:31:35.599683+0000","last_became_active":"2024-08-19T01:30:51.551795+0000","last_became_peered":"2024-08-19T01:30:51.551795+0000","last_unstale":"2024-08-19T01:31:35.599683+0000","last_undegraded":"2024-08-19T01:31:35.599683+0000","last_fullsized":"2024-08-19T01:31:35.599683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:22:54.705359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868723+0000","last_change":"2024-08-19T01:30:49.536659+0000","last_active":"2024-08-19T01:31:35.868723+0000","last_peered":"2024-08-19T01:31:35.868723+0000","last_clean":"2024-08-19T01:31:35.868723+0000","last_became_active":"2024-08-19T01:30:49.536366+0000","last_became_peered":"2024-08-19T01:30:49.536366+0000","last_unstale":"2024-08-19T01:31:35.868723+0000","last_undegraded":"2024-08-19T01:31:35.868723+0000","last_fullsized":"2024-08-19T01:31:35.868723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:58:52.547461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595100+0000","last_change":"2024-08-19T01:30:58.593724+0000","last_active":"2024-08-19T01:31:35.595100+0000","last_peered":"2024-08-19T01:31:35.595100+0000","last_clean":"2024-08-19T01:31:35.595100+0000","last_became_active":"2024-08-19T01:30:58.593383+0000","last_became_peered":"2024-08-19T01:30:58.593383+0000","last_unstale":"2024-08-19T01:31:35.595100+0000","last_undegraded":"2024-08-19T01:31:35.595100+0000","last_fullsized":"2024-08-19T01:31:35.595100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:54:26.424139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603836+0000","last_change":"2024-08-19T01:30:51.557682+0000","last_active":"2024-08-19T01:31:35.603836+0000","last_peered":"2024-08-19T01:31:35.603836+0000","last_clean":"2024-08-19T01:31:35.603836+0000","last_became_active":"2024-08-19T01:30:51.557351+0000","last_became_peered":"2024-08-19T01:30:51.557351+0000","last_unstale":"2024-08-19T01:31:35.603836+0000","last_undegraded":"2024-08-19T01:31:35.603836+0000","last_fullsized":"2024-08-19T01:31:35.603836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:41.484955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595958+0000","last_change":"2024-08-19T01:30:49.575996+0000","last_active":"2024-08-19T01:31:35.595958+0000","last_peered":"2024-08-19T01:31:35.595958+0000","last_clean":"2024-08-19T01:31:35.595958+0000","last_became_active":"2024-08-19T01:30:49.575607+0000","last_became_peered":"2024-08-19T01:30:49.575607+0000","last_unstale":"2024-08-19T01:31:35.595958+0000","last_undegraded":"2024-08-19T01:31:35.595958+0000","last_fullsized":"2024-08-19T01:31:35.595958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:20:33.467100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600113+0000","last_change":"2024-08-19T01:30:58.592629+0000","last_active":"2024-08-19T01:31:35.600113+0000","last_peered":"2024-08-19T01:31:35.600113+0000","last_clean":"2024-08-19T01:31:35.600113+0000","last_became_active":"2024-08-19T01:30:58.592122+0000","last_became_peered":"2024-08-19T01:30:58.592122+0000","last_unstale":"2024-08-19T01:31:35.600113+0000","last_undegraded":"2024-08-19T01:31:35.600113+0000","last_fullsized":"2024-08-19T01:31:35.600113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:19:09.846829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.746416+0000","last_change":"2024-08-19T01:31:35.746416+0000","last_active":"2024-08-19T01:31:35.746416+0000","last_peered":"2024-08-19T01:31:35.746416+0000","last_clean":"2024-08-19T01:31:35.746416+0000","last_became_active":"2024-08-19T01:31:35.746110+0000","last_became_peered":"2024-08-19T01:31:35.746110+0000","last_unstale":"2024-08-19T01:31:35.746416+0000","last_undegraded":"2024-08-19T01:31:35.746416+0000","last_fullsized":"2024-08-19T01:31:35.746416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:29:29.789785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600943+0000","last_change":"2024-08-19T01:30:49.711366+0000","last_active":"2024-08-19T01:31:35.600943+0000","last_peered":"2024-08-19T01:31:35.600943+0000","last_clean":"2024-08-19T01:31:35.600943+0000","last_became_active":"2024-08-19T01:30:49.711078+0000","last_became_peered":"2024-08-19T01:30:49.711078+0000","last_unstale":"2024-08-19T01:31:35.600943+0000","last_undegraded":"2024-08-19T01:31:35.600943+0000","last_fullsized":"2024-08-19T01:31:35.600943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:11:16.767867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597334+0000","last_change":"2024-08-19T01:30:58.591845+0000","last_active":"2024-08-19T01:31:35.597334+0000","last_peered":"2024-08-19T01:31:35.597334+0000","last_clean":"2024-08-19T01:31:35.597334+0000","last_became_active":"2024-08-19T01:30:58.591453+0000","last_became_peered":"2024-08-19T01:30:58.591453+0000","last_unstale":"2024-08-19T01:31:35.597334+0000","last_undegraded":"2024-08-19T01:31:35.597334+0000","last_fullsized":"2024-08-19T01:31:35.597334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:37:09.521514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.869013+0000","last_change":"2024-08-19T01:30:51.563156+0000","last_active":"2024-08-19T01:31:35.869013+0000","last_peered":"2024-08-19T01:31:35.869013+0000","last_clean":"2024-08-19T01:31:35.869013+0000","last_became_active":"2024-08-19T01:30:51.562860+0000","last_became_peered":"2024-08-19T01:30:51.562860+0000","last_unstale":"2024-08-19T01:31:35.869013+0000","last_undegraded":"2024-08-19T01:31:35.869013+0000","last_fullsized":"2024-08-19T01:31:35.869013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:54:11.768833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:48.559439+0000","last_change":"2024-08-19T01:30:49.536648+0000","last_active":"2024-08-19T01:31:48.559439+0000","last_peered":"2024-08-19T01:31:48.559439+0000","last_clean":"2024-08-19T01:31:48.559439+0000","last_became_active":"2024-08-19T01:30:49.536080+0000","last_became_peered":"2024-08-19T01:30:49.536080+0000","last_unstale":"2024-08-19T01:31:48.559439+0000","last_undegraded":"2024-08-19T01:31:48.559439+0000","last_fullsized":"2024-08-19T01:31:48.559439+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:03:46.804252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868291+0000","last_change":"2024-08-19T01:30:58.592253+0000","last_active":"2024-08-19T01:31:35.868291+0000","last_peered":"2024-08-19T01:31:35.868291+0000","last_clean":"2024-08-19T01:31:35.868291+0000","last_became_active":"2024-08-19T01:30:58.591911+0000","last_became_peered":"2024-08-19T01:30:58.591911+0000","last_unstale":"2024-08-19T01:31:35.868291+0000","last_undegraded":"2024-08-19T01:31:35.868291+0000","last_fullsized":"2024-08-19T01:31:35.868291+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:02:27.112597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745492+0000","last_change":"2024-08-19T01:30:51.547607+0000","last_active":"2024-08-19T01:31:35.745492+0000","last_peered":"2024-08-19T01:31:35.745492+0000","last_clean":"2024-08-19T01:31:35.745492+0000","last_became_active":"2024-08-19T01:30:51.547316+0000","last_became_peered":"2024-08-19T01:30:51.547316+0000","last_unstale":"2024-08-19T01:31:35.745492+0000","last_undegraded":"2024-08-19T01:31:35.745492+0000","last_fullsized":"2024-08-19T01:31:35.745492+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:48:58.300508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604727+0000","last_change":"2024-08-19T01:30:49.576942+0000","last_active":"2024-08-19T01:31:35.604727+0000","last_peered":"2024-08-19T01:31:35.604727+0000","last_clean":"2024-08-19T01:31:35.604727+0000","last_became_active":"2024-08-19T01:30:49.576528+0000","last_became_peered":"2024-08-19T01:30:49.576528+0000","last_unstale":"2024-08-19T01:31:35.604727+0000","last_undegraded":"2024-08-19T01:31:35.604727+0000","last_fullsized":"2024-08-19T01:31:35.604727+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:23:53.259395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600274+0000","last_change":"2024-08-19T01:30:58.591577+0000","last_active":"2024-08-19T01:31:35.600274+0000","last_peered":"2024-08-19T01:31:35.600274+0000","last_clean":"2024-08-19T01:31:35.600274+0000","last_became_active":"2024-08-19T01:30:58.591181+0000","last_became_peered":"2024-08-19T01:30:58.591181+0000","last_unstale":"2024-08-19T01:31:35.600274+0000","last_undegraded":"2024-08-19T01:31:35.600274+0000","last_fullsized":"2024-08-19T01:31:35.600274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:02:59.682147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598931+0000","last_change":"2024-08-19T01:30:51.548556+0000","last_active":"2024-08-19T01:31:35.598931+0000","last_peered":"2024-08-19T01:31:35.598931+0000","last_clean":"2024-08-19T01:31:35.598931+0000","last_became_active":"2024-08-19T01:30:51.547838+0000","last_became_peered":"2024-08-19T01:30:51.547838+0000","last_unstale":"2024-08-19T01:31:35.598931+0000","last_undegraded":"2024-08-19T01:31:35.598931+0000","last_fullsized":"2024-08-19T01:31:35.598931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:08:49.671964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595172+0000","last_change":"2024-08-19T01:30:49.576799+0000","last_active":"2024-08-19T01:31:35.595172+0000","last_peered":"2024-08-19T01:31:35.595172+0000","last_clean":"2024-08-19T01:31:35.595172+0000","last_became_active":"2024-08-19T01:30:49.576385+0000","last_became_peered":"2024-08-19T01:30:49.576385+0000","last_unstale":"2024-08-19T01:31:35.595172+0000","last_undegraded":"2024-08-19T01:31:35.595172+0000","last_fullsized":"2024-08-19T01:31:35.595172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:34:18.287518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603305+0000","last_change":"2024-08-19T01:30:58.593703+0000","last_active":"2024-08-19T01:31:35.603305+0000","last_peered":"2024-08-19T01:31:35.603305+0000","last_clean":"2024-08-19T01:31:35.603305+0000","last_became_active":"2024-08-19T01:30:58.593303+0000","last_became_peered":"2024-08-19T01:30:58.593303+0000","last_unstale":"2024-08-19T01:31:35.603305+0000","last_undegraded":"2024-08-19T01:31:35.603305+0000","last_fullsized":"2024-08-19T01:31:35.603305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:12:43.359262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601173+0000","last_change":"2024-08-19T01:31:35.601173+0000","last_active":"2024-08-19T01:31:35.601173+0000","last_peered":"2024-08-19T01:31:35.601173+0000","last_clean":"2024-08-19T01:31:35.601173+0000","last_became_active":"2024-08-19T01:31:35.600742+0000","last_became_peered":"2024-08-19T01:31:35.600742+0000","last_unstale":"2024-08-19T01:31:35.601173+0000","last_undegraded":"2024-08-19T01:31:35.601173+0000","last_fullsized":"2024-08-19T01:31:35.601173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:36:48.977124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604573+0000","last_change":"2024-08-19T01:30:49.540070+0000","last_active":"2024-08-19T01:31:35.604573+0000","last_peered":"2024-08-19T01:31:35.604573+0000","last_clean":"2024-08-19T01:31:35.604573+0000","last_became_active":"2024-08-19T01:30:49.539706+0000","last_became_peered":"2024-08-19T01:30:49.539706+0000","last_unstale":"2024-08-19T01:31:35.604573+0000","last_undegraded":"2024-08-19T01:31:35.604573+0000","last_fullsized":"2024-08-19T01:31:35.604573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:14:15.813670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745319+0000","last_change":"2024-08-19T01:30:58.590766+0000","last_active":"2024-08-19T01:31:35.745319+0000","last_peered":"2024-08-19T01:31:35.745319+0000","last_clean":"2024-08-19T01:31:35.745319+0000","last_became_active":"2024-08-19T01:30:58.590415+0000","last_became_peered":"2024-08-19T01:30:58.590415+0000","last_unstale":"2024-08-19T01:31:35.745319+0000","last_undegraded":"2024-08-19T01:31:35.745319+0000","last_fullsized":"2024-08-19T01:31:35.745319+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:48:59.261086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.602761+0000","last_change":"2024-08-19T01:31:35.602332+0000","last_active":"2024-08-19T01:31:35.602761+0000","last_peered":"2024-08-19T01:31:35.602761+0000","last_clean":"2024-08-19T01:31:35.602761+0000","last_became_active":"2024-08-19T01:31:35.602007+0000","last_became_peered":"2024-08-19T01:31:35.602007+0000","last_unstale":"2024-08-19T01:31:35.602761+0000","last_undegraded":"2024-08-19T01:31:35.602761+0000","last_fullsized":"2024-08-19T01:31:35.602761+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:17:47.694567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:23.671848+0000","last_change":"2024-08-19T01:30:49.575995+0000","last_active":"2024-08-19T01:32:23.671848+0000","last_peered":"2024-08-19T01:32:23.671848+0000","last_clean":"2024-08-19T01:32:23.671848+0000","last_became_active":"2024-08-19T01:30:49.575249+0000","last_became_peered":"2024-08-19T01:30:49.575249+0000","last_unstale":"2024-08-19T01:32:23.671848+0000","last_undegraded":"2024-08-19T01:32:23.671848+0000","last_fullsized":"2024-08-19T01:32:23.671848+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:39:05.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745536+0000","last_change":"2024-08-19T01:30:58.598812+0000","last_active":"2024-08-19T01:31:35.745536+0000","last_peered":"2024-08-19T01:31:35.745536+0000","last_clean":"2024-08-19T01:31:35.745536+0000","last_became_active":"2024-08-19T01:30:58.598434+0000","last_became_peered":"2024-08-19T01:30:58.598434+0000","last_unstale":"2024-08-19T01:31:35.745536+0000","last_undegraded":"2024-08-19T01:31:35.745536+0000","last_fullsized":"2024-08-19T01:31:35.745536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:15:19.381503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744921+0000","last_change":"2024-08-19T01:30:51.549158+0000","last_active":"2024-08-19T01:31:35.744921+0000","last_peered":"2024-08-19T01:31:35.744921+0000","last_clean":"2024-08-19T01:31:35.744921+0000","last_became_active":"2024-08-19T01:30:51.548854+0000","last_became_peered":"2024-08-19T01:30:51.548854+0000","last_unstale":"2024-08-19T01:31:35.744921+0000","last_undegraded":"2024-08-19T01:31:35.744921+0000","last_fullsized":"2024-08-19T01:31:35.744921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:35.661758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604219+0000","last_change":"2024-08-19T01:30:49.538282+0000","last_active":"2024-08-19T01:31:35.604219+0000","last_peered":"2024-08-19T01:31:35.604219+0000","last_clean":"2024-08-19T01:31:35.604219+0000","last_became_active":"2024-08-19T01:30:49.537918+0000","last_became_peered":"2024-08-19T01:30:49.537918+0000","last_unstale":"2024-08-19T01:31:35.604219+0000","last_undegraded":"2024-08-19T01:31:35.604219+0000","last_fullsized":"2024-08-19T01:31:35.604219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:27.886958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601012+0000","last_change":"2024-08-19T01:30:58.591153+0000","last_active":"2024-08-19T01:31:35.601012+0000","last_peered":"2024-08-19T01:31:35.601012+0000","last_clean":"2024-08-19T01:31:35.601012+0000","last_became_active":"2024-08-19T01:30:58.590738+0000","last_became_peered":"2024-08-19T01:30:58.590738+0000","last_unstale":"2024-08-19T01:31:35.601012+0000","last_undegraded":"2024-08-19T01:31:35.601012+0000","last_fullsized":"2024-08-19T01:31:35.601012+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:29.073825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868743+0000","last_change":"2024-08-19T01:30:51.554878+0000","last_active":"2024-08-19T01:31:35.868743+0000","last_peered":"2024-08-19T01:31:35.868743+0000","last_clean":"2024-08-19T01:31:35.868743+0000","last_became_active":"2024-08-19T01:30:51.553743+0000","last_became_peered":"2024-08-19T01:30:51.553743+0000","last_unstale":"2024-08-19T01:31:35.868743+0000","last_undegraded":"2024-08-19T01:31:35.868743+0000","last_fullsized":"2024-08-19T01:31:35.868743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:08:41.932641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600867+0000","last_change":"2024-08-19T01:30:49.542043+0000","last_active":"2024-08-19T01:31:35.600867+0000","last_peered":"2024-08-19T01:31:35.600867+0000","last_clean":"2024-08-19T01:31:35.600867+0000","last_became_active":"2024-08-19T01:30:49.541551+0000","last_became_peered":"2024-08-19T01:30:49.541551+0000","last_unstale":"2024-08-19T01:31:35.600867+0000","last_undegraded":"2024-08-19T01:31:35.600867+0000","last_fullsized":"2024-08-19T01:31:35.600867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:46:14.781516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":364,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604258+0000","last_change":"2024-08-19T01:29:41.574391+0000","last_active":"2024-08-19T01:31:35.604258+0000","last_peered":"2024-08-19T01:31:35.604258+0000","last_clean":"2024-08-19T01:31:35.604258+0000","last_became_active":"2024-08-19T01:29:41.573885+0000","last_became_peered":"2024-08-19T01:29:41.573885+0000","last_unstale":"2024-08-19T01:31:35.604258+0000","last_undegraded":"2024-08-19T01:31:35.604258+0000","last_fullsized":"2024-08-19T01:31:35.604258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_clean_scrub_stamp":"2024-08-19T01:26:37.926225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:55:38.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:58.598415+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:58.598086+0000","last_became_peered":"2024-08-19T01:30:58.598086+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:21:37.338869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605022+0000","last_change":"2024-08-19T01:30:51.560407+0000","last_active":"2024-08-19T01:31:35.605022+0000","last_peered":"2024-08-19T01:31:35.605022+0000","last_clean":"2024-08-19T01:31:35.605022+0000","last_became_active":"2024-08-19T01:30:51.559881+0000","last_became_peered":"2024-08-19T01:30:51.559881+0000","last_unstale":"2024-08-19T01:31:35.605022+0000","last_undegraded":"2024-08-19T01:31:35.605022+0000","last_fullsized":"2024-08-19T01:31:35.605022+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:24:37.928254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597862+0000","last_change":"2024-08-19T01:30:49.711171+0000","last_active":"2024-08-19T01:31:35.597862+0000","last_peered":"2024-08-19T01:31:35.597862+0000","last_clean":"2024-08-19T01:31:35.597862+0000","last_became_active":"2024-08-19T01:30:49.710707+0000","last_became_peered":"2024-08-19T01:30:49.710707+0000","last_unstale":"2024-08-19T01:31:35.597862+0000","last_undegraded":"2024-08-19T01:31:35.597862+0000","last_fullsized":"2024-08-19T01:31:35.597862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:58:20.924502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595955+0000","last_change":"2024-08-19T01:30:58.590829+0000","last_active":"2024-08-19T01:31:35.595955+0000","last_peered":"2024-08-19T01:31:35.595955+0000","last_clean":"2024-08-19T01:31:35.595955+0000","last_became_active":"2024-08-19T01:30:58.590460+0000","last_became_peered":"2024-08-19T01:30:58.590460+0000","last_unstale":"2024-08-19T01:31:35.595955+0000","last_undegraded":"2024-08-19T01:31:35.595955+0000","last_fullsized":"2024-08-19T01:31:35.595955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:04.118550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604645+0000","last_change":"2024-08-19T01:30:51.564990+0000","last_active":"2024-08-19T01:31:35.604645+0000","last_peered":"2024-08-19T01:31:35.604645+0000","last_clean":"2024-08-19T01:31:35.604645+0000","last_became_active":"2024-08-19T01:30:51.564653+0000","last_became_peered":"2024-08-19T01:30:51.564653+0000","last_unstale":"2024-08-19T01:31:35.604645+0000","last_undegraded":"2024-08-19T01:31:35.604645+0000","last_fullsized":"2024-08-19T01:31:35.604645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:15:41.154376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601128+0000","last_change":"2024-08-19T01:30:49.537637+0000","last_active":"2024-08-19T01:31:35.601128+0000","last_peered":"2024-08-19T01:31:35.601128+0000","last_clean":"2024-08-19T01:31:35.601128+0000","last_became_active":"2024-08-19T01:30:49.537277+0000","last_became_peered":"2024-08-19T01:30:49.537277+0000","last_unstale":"2024-08-19T01:31:35.601128+0000","last_undegraded":"2024-08-19T01:31:35.601128+0000","last_fullsized":"2024-08-19T01:31:35.601128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:13.547597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598003+0000","last_change":"2024-08-19T01:30:58.594675+0000","last_active":"2024-08-19T01:31:35.598003+0000","last_peered":"2024-08-19T01:31:35.598003+0000","last_clean":"2024-08-19T01:31:35.598003+0000","last_became_active":"2024-08-19T01:30:58.594323+0000","last_became_peered":"2024-08-19T01:30:58.594323+0000","last_unstale":"2024-08-19T01:31:35.598003+0000","last_undegraded":"2024-08-19T01:31:35.598003+0000","last_fullsized":"2024-08-19T01:31:35.598003+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:21:05.447106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605468+0000","last_change":"2024-08-19T01:30:51.565014+0000","last_active":"2024-08-19T01:31:35.605468+0000","last_peered":"2024-08-19T01:31:35.605468+0000","last_clean":"2024-08-19T01:31:35.605468+0000","last_became_active":"2024-08-19T01:30:51.564672+0000","last_became_peered":"2024-08-19T01:30:51.564672+0000","last_unstale":"2024-08-19T01:31:35.605468+0000","last_undegraded":"2024-08-19T01:31:35.605468+0000","last_fullsized":"2024-08-19T01:31:35.605468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:56:07.404042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745571+0000","last_change":"2024-08-19T01:30:49.538423+0000","last_active":"2024-08-19T01:31:35.745571+0000","last_peered":"2024-08-19T01:31:35.745571+0000","last_clean":"2024-08-19T01:31:35.745571+0000","last_became_active":"2024-08-19T01:30:49.538049+0000","last_became_peered":"2024-08-19T01:30:49.538049+0000","last_unstale":"2024-08-19T01:31:35.745571+0000","last_undegraded":"2024-08-19T01:31:35.745571+0000","last_fullsized":"2024-08-19T01:31:35.745571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:27:41.449742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599789+0000","last_change":"2024-08-19T01:30:58.590963+0000","last_active":"2024-08-19T01:31:35.599789+0000","last_peered":"2024-08-19T01:31:35.599789+0000","last_clean":"2024-08-19T01:31:35.599789+0000","last_became_active":"2024-08-19T01:30:58.590584+0000","last_became_peered":"2024-08-19T01:30:58.590584+0000","last_unstale":"2024-08-19T01:31:35.599789+0000","last_undegraded":"2024-08-19T01:31:35.599789+0000","last_fullsized":"2024-08-19T01:31:35.599789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:37.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599809+0000","last_change":"2024-08-19T01:30:51.561298+0000","last_active":"2024-08-19T01:31:35.599809+0000","last_peered":"2024-08-19T01:31:35.599809+0000","last_clean":"2024-08-19T01:31:35.599809+0000","last_became_active":"2024-08-19T01:30:51.560919+0000","last_became_peered":"2024-08-19T01:30:51.560919+0000","last_unstale":"2024-08-19T01:31:35.599809+0000","last_undegraded":"2024-08-19T01:31:35.599809+0000","last_fullsized":"2024-08-19T01:31:35.599809+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:04:39.808838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604897+0000","last_change":"2024-08-19T01:30:49.541968+0000","last_active":"2024-08-19T01:31:35.604897+0000","last_peered":"2024-08-19T01:31:35.604897+0000","last_clean":"2024-08-19T01:31:35.604897+0000","last_became_active":"2024-08-19T01:30:49.541512+0000","last_became_peered":"2024-08-19T01:30:49.541512+0000","last_unstale":"2024-08-19T01:31:35.604897+0000","last_undegraded":"2024-08-19T01:31:35.604897+0000","last_fullsized":"2024-08-19T01:31:35.604897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:34:28.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600022+0000","last_change":"2024-08-19T01:30:58.594776+0000","last_active":"2024-08-19T01:31:35.600022+0000","last_peered":"2024-08-19T01:31:35.600022+0000","last_clean":"2024-08-19T01:31:35.600022+0000","last_became_active":"2024-08-19T01:30:58.594414+0000","last_became_peered":"2024-08-19T01:30:58.594414+0000","last_unstale":"2024-08-19T01:31:35.600022+0000","last_undegraded":"2024-08-19T01:31:35.600022+0000","last_fullsized":"2024-08-19T01:31:35.600022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:32:20.121504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600072+0000","last_change":"2024-08-19T01:30:51.556696+0000","last_active":"2024-08-19T01:31:35.600072+0000","last_peered":"2024-08-19T01:31:35.600072+0000","last_clean":"2024-08-19T01:31:35.600072+0000","last_became_active":"2024-08-19T01:30:51.555873+0000","last_became_peered":"2024-08-19T01:30:51.555873+0000","last_unstale":"2024-08-19T01:31:35.600072+0000","last_undegraded":"2024-08-19T01:31:35.600072+0000","last_fullsized":"2024-08-19T01:31:35.600072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:50:27.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603398+0000","last_change":"2024-08-19T01:30:49.576925+0000","last_active":"2024-08-19T01:31:35.603398+0000","last_peered":"2024-08-19T01:31:35.603398+0000","last_clean":"2024-08-19T01:31:35.603398+0000","last_became_active":"2024-08-19T01:30:49.576509+0000","last_became_peered":"2024-08-19T01:30:49.576509+0000","last_unstale":"2024-08-19T01:31:35.603398+0000","last_undegraded":"2024-08-19T01:31:35.603398+0000","last_fullsized":"2024-08-19T01:31:35.603398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:43:01.887020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745511+0000","last_change":"2024-08-19T01:30:58.590940+0000","last_active":"2024-08-19T01:31:35.745511+0000","last_peered":"2024-08-19T01:31:35.745511+0000","last_clean":"2024-08-19T01:31:35.745511+0000","last_became_active":"2024-08-19T01:30:58.590565+0000","last_became_peered":"2024-08-19T01:30:58.590565+0000","last_unstale":"2024-08-19T01:31:35.745511+0000","last_undegraded":"2024-08-19T01:31:35.745511+0000","last_fullsized":"2024-08-19T01:31:35.745511+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:38:31.345290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603937+0000","last_change":"2024-08-19T01:30:51.559622+0000","last_active":"2024-08-19T01:31:35.603937+0000","last_peered":"2024-08-19T01:31:35.603937+0000","last_clean":"2024-08-19T01:31:35.603937+0000","last_became_active":"2024-08-19T01:30:51.559263+0000","last_became_peered":"2024-08-19T01:30:51.559263+0000","last_unstale":"2024-08-19T01:31:35.603937+0000","last_undegraded":"2024-08-19T01:31:35.603937+0000","last_fullsized":"2024-08-19T01:31:35.603937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:11:25.758915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:23.672309+0000","last_change":"2024-08-19T01:30:49.540102+0000","last_active":"2024-08-19T01:32:23.672309+0000","last_peered":"2024-08-19T01:32:23.672309+0000","last_clean":"2024-08-19T01:32:23.672309+0000","last_became_active":"2024-08-19T01:30:49.539740+0000","last_became_peered":"2024-08-19T01:30:49.539740+0000","last_unstale":"2024-08-19T01:32:23.672309+0000","last_undegraded":"2024-08-19T01:32:23.672309+0000","last_fullsized":"2024-08-19T01:32:23.672309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:47:21.684925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595191+0000","last_change":"2024-08-19T01:30:58.595748+0000","last_active":"2024-08-19T01:31:35.595191+0000","last_peered":"2024-08-19T01:31:35.595191+0000","last_clean":"2024-08-19T01:31:35.595191+0000","last_became_active":"2024-08-19T01:30:58.595396+0000","last_became_peered":"2024-08-19T01:30:58.595396+0000","last_unstale":"2024-08-19T01:31:35.595191+0000","last_undegraded":"2024-08-19T01:31:35.595191+0000","last_fullsized":"2024-08-19T01:31:35.595191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:58:29.113687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604652+0000","last_change":"2024-08-19T01:30:51.559159+0000","last_active":"2024-08-19T01:31:35.604652+0000","last_peered":"2024-08-19T01:31:35.604652+0000","last_clean":"2024-08-19T01:31:35.604652+0000","last_became_active":"2024-08-19T01:30:51.558226+0000","last_became_peered":"2024-08-19T01:30:51.558226+0000","last_unstale":"2024-08-19T01:31:35.604652+0000","last_undegraded":"2024-08-19T01:31:35.604652+0000","last_fullsized":"2024-08-19T01:31:35.604652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:12:54.447301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600514+0000","last_change":"2024-08-19T01:30:49.540806+0000","last_active":"2024-08-19T01:31:35.600514+0000","last_peered":"2024-08-19T01:31:35.600514+0000","last_clean":"2024-08-19T01:31:35.600514+0000","last_became_active":"2024-08-19T01:30:49.540439+0000","last_became_peered":"2024-08-19T01:30:49.540439+0000","last_unstale":"2024-08-19T01:31:35.600514+0000","last_undegraded":"2024-08-19T01:31:35.600514+0000","last_fullsized":"2024-08-19T01:31:35.600514+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:40.088842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599613+0000","last_change":"2024-08-19T01:30:58.592661+0000","last_active":"2024-08-19T01:31:35.599613+0000","last_peered":"2024-08-19T01:31:35.599613+0000","last_clean":"2024-08-19T01:31:35.599613+0000","last_became_active":"2024-08-19T01:30:58.592066+0000","last_became_peered":"2024-08-19T01:30:58.592066+0000","last_unstale":"2024-08-19T01:31:35.599613+0000","last_undegraded":"2024-08-19T01:31:35.599613+0000","last_fullsized":"2024-08-19T01:31:35.599613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:35.039682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745099+0000","last_change":"2024-08-19T01:30:51.558497+0000","last_active":"2024-08-19T01:31:35.745099+0000","last_peered":"2024-08-19T01:31:35.745099+0000","last_clean":"2024-08-19T01:31:35.745099+0000","last_became_active":"2024-08-19T01:30:51.558185+0000","last_became_peered":"2024-08-19T01:30:51.558185+0000","last_unstale":"2024-08-19T01:31:35.745099+0000","last_undegraded":"2024-08-19T01:31:35.745099+0000","last_fullsized":"2024-08-19T01:31:35.745099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:30:42.766269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597725+0000","last_change":"2024-08-19T01:30:49.539112+0000","last_active":"2024-08-19T01:31:35.597725+0000","last_peered":"2024-08-19T01:31:35.597725+0000","last_clean":"2024-08-19T01:31:35.597725+0000","last_became_active":"2024-08-19T01:30:49.538798+0000","last_became_peered":"2024-08-19T01:30:49.538798+0000","last_unstale":"2024-08-19T01:31:35.597725+0000","last_undegraded":"2024-08-19T01:31:35.597725+0000","last_fullsized":"2024-08-19T01:31:35.597725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:10:58.977041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600778+0000","last_change":"2024-08-19T01:30:58.590727+0000","last_active":"2024-08-19T01:31:35.600778+0000","last_peered":"2024-08-19T01:31:35.600778+0000","last_clean":"2024-08-19T01:31:35.600778+0000","last_became_active":"2024-08-19T01:30:58.590208+0000","last_became_peered":"2024-08-19T01:30:58.590208+0000","last_unstale":"2024-08-19T01:31:35.600778+0000","last_undegraded":"2024-08-19T01:31:35.600778+0000","last_fullsized":"2024-08-19T01:31:35.600778+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:36.422915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604177+0000","last_change":"2024-08-19T01:30:51.565113+0000","last_active":"2024-08-19T01:31:35.604177+0000","last_peered":"2024-08-19T01:31:35.604177+0000","last_clean":"2024-08-19T01:31:35.604177+0000","last_became_active":"2024-08-19T01:30:51.564769+0000","last_became_peered":"2024-08-19T01:30:51.564769+0000","last_unstale":"2024-08-19T01:31:35.604177+0000","last_undegraded":"2024-08-19T01:31:35.604177+0000","last_fullsized":"2024-08-19T01:31:35.604177+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:13:28.373114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'32","reported_seq":201,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:27.676482+0000","last_change":"2024-08-19T01:31:35.610319+0000","last_active":"2024-08-19T01:32:27.676482+0000","last_peered":"2024-08-19T01:32:27.676482+0000","last_clean":"2024-08-19T01:32:27.676482+0000","last_became_active":"2024-08-19T01:31:35.602527+0000","last_became_peered":"2024-08-19T01:31:35.602527+0000","last_unstale":"2024-08-19T01:32:27.676482+0000","last_undegraded":"2024-08-19T01:32:27.676482+0000","last_fullsized":"2024-08-19T01:32:27.676482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:50:05.511332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62070,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":113,"num_read_kb":114,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3841,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594820+0000","last_change":"2024-08-19T01:30:58.594836+0000","last_active":"2024-08-19T01:31:35.594820+0000","last_peered":"2024-08-19T01:31:35.594820+0000","last_clean":"2024-08-19T01:31:35.594820+0000","last_became_active":"2024-08-19T01:30:58.594441+0000","last_became_peered":"2024-08-19T01:30:58.594441+0000","last_unstale":"2024-08-19T01:31:35.594820+0000","last_undegraded":"2024-08-19T01:31:35.594820+0000","last_fullsized":"2024-08-19T01:31:35.594820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:29:16.482181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596226+0000","last_change":"2024-08-19T01:30:51.562567+0000","last_active":"2024-08-19T01:31:35.596226+0000","last_peered":"2024-08-19T01:31:35.596226+0000","last_clean":"2024-08-19T01:31:35.596226+0000","last_became_active":"2024-08-19T01:30:51.562211+0000","last_became_peered":"2024-08-19T01:30:51.562211+0000","last_unstale":"2024-08-19T01:31:35.596226+0000","last_undegraded":"2024-08-19T01:31:35.596226+0000","last_fullsized":"2024-08-19T01:31:35.596226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:58.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601097+0000","last_change":"2024-08-19T01:30:49.575957+0000","last_active":"2024-08-19T01:31:35.601097+0000","last_peered":"2024-08-19T01:31:35.601097+0000","last_clean":"2024-08-19T01:31:35.601097+0000","last_became_active":"2024-08-19T01:30:49.575570+0000","last_became_peered":"2024-08-19T01:30:49.575570+0000","last_unstale":"2024-08-19T01:31:35.601097+0000","last_undegraded":"2024-08-19T01:31:35.601097+0000","last_fullsized":"2024-08-19T01:31:35.601097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:47.038113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599852+0000","last_change":"2024-08-19T01:30:58.589830+0000","last_active":"2024-08-19T01:31:35.599852+0000","last_peered":"2024-08-19T01:31:35.599852+0000","last_clean":"2024-08-19T01:31:35.599852+0000","last_became_active":"2024-08-19T01:30:58.589292+0000","last_became_peered":"2024-08-19T01:30:58.589292+0000","last_unstale":"2024-08-19T01:31:35.599852+0000","last_undegraded":"2024-08-19T01:31:35.599852+0000","last_fullsized":"2024-08-19T01:31:35.599852+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:14.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.606449+0000","last_change":"2024-08-19T01:31:35.606449+0000","last_active":"2024-08-19T01:31:35.606449+0000","last_peered":"2024-08-19T01:31:35.606449+0000","last_clean":"2024-08-19T01:31:35.606449+0000","last_became_active":"2024-08-19T01:31:35.606105+0000","last_became_peered":"2024-08-19T01:31:35.606105+0000","last_unstale":"2024-08-19T01:31:35.606449+0000","last_undegraded":"2024-08-19T01:31:35.606449+0000","last_fullsized":"2024-08-19T01:31:35.606449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:36:21.834763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.605342+0000","last_change":"2024-08-19T01:30:49.542441+0000","last_active":"2024-08-19T01:31:35.605342+0000","last_peered":"2024-08-19T01:31:35.605342+0000","last_clean":"2024-08-19T01:31:35.605342+0000","last_became_active":"2024-08-19T01:30:49.542106+0000","last_became_peered":"2024-08-19T01:30:49.542106+0000","last_unstale":"2024-08-19T01:31:35.605342+0000","last_undegraded":"2024-08-19T01:31:35.605342+0000","last_fullsized":"2024-08-19T01:31:35.605342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:21:01.956282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594574+0000","last_change":"2024-08-19T01:30:58.588899+0000","last_active":"2024-08-19T01:31:35.594574+0000","last_peered":"2024-08-19T01:31:35.594574+0000","last_clean":"2024-08-19T01:31:35.594574+0000","last_became_active":"2024-08-19T01:30:58.588521+0000","last_became_peered":"2024-08-19T01:30:58.588521+0000","last_unstale":"2024-08-19T01:31:35.594574+0000","last_undegraded":"2024-08-19T01:31:35.594574+0000","last_fullsized":"2024-08-19T01:31:35.594574+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:43:59.485456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596747+0000","last_change":"2024-08-19T01:30:51.557248+0000","last_active":"2024-08-19T01:31:35.596747+0000","last_peered":"2024-08-19T01:31:35.596747+0000","last_clean":"2024-08-19T01:31:35.596747+0000","last_became_active":"2024-08-19T01:30:51.556448+0000","last_became_peered":"2024-08-19T01:30:51.556448+0000","last_unstale":"2024-08-19T01:31:35.596747+0000","last_undegraded":"2024-08-19T01:31:35.596747+0000","last_fullsized":"2024-08-19T01:31:35.596747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:19:19.587955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:26.675240+0000","last_change":"2024-08-19T01:30:49.540360+0000","last_active":"2024-08-19T01:32:26.675240+0000","last_peered":"2024-08-19T01:32:26.675240+0000","last_clean":"2024-08-19T01:32:26.675240+0000","last_became_active":"2024-08-19T01:30:49.539993+0000","last_became_peered":"2024-08-19T01:30:49.539993+0000","last_unstale":"2024-08-19T01:32:26.675240+0000","last_undegraded":"2024-08-19T01:32:26.675240+0000","last_fullsized":"2024-08-19T01:32:26.675240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:05:28.308441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868467+0000","last_change":"2024-08-19T01:30:58.592387+0000","last_active":"2024-08-19T01:31:35.868467+0000","last_peered":"2024-08-19T01:31:35.868467+0000","last_clean":"2024-08-19T01:31:35.868467+0000","last_became_active":"2024-08-19T01:30:58.592031+0000","last_became_peered":"2024-08-19T01:30:58.592031+0000","last_unstale":"2024-08-19T01:31:35.868467+0000","last_undegraded":"2024-08-19T01:31:35.868467+0000","last_fullsized":"2024-08-19T01:31:35.868467+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:31:23.159012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598112+0000","last_change":"2024-08-19T01:30:58.595650+0000","last_active":"2024-08-19T01:31:35.598112+0000","last_peered":"2024-08-19T01:31:35.598112+0000","last_clean":"2024-08-19T01:31:35.598112+0000","last_became_active":"2024-08-19T01:30:58.595304+0000","last_became_peered":"2024-08-19T01:30:58.595304+0000","last_unstale":"2024-08-19T01:31:35.598112+0000","last_undegraded":"2024-08-19T01:31:35.598112+0000","last_fullsized":"2024-08-19T01:31:35.598112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:23:53.517355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.594771+0000","last_change":"2024-08-19T01:30:51.548683+0000","last_active":"2024-08-19T01:31:35.594771+0000","last_peered":"2024-08-19T01:31:35.594771+0000","last_clean":"2024-08-19T01:31:35.594771+0000","last_became_active":"2024-08-19T01:30:51.548382+0000","last_became_peered":"2024-08-19T01:30:51.548382+0000","last_unstale":"2024-08-19T01:31:35.594771+0000","last_undegraded":"2024-08-19T01:31:35.594771+0000","last_fullsized":"2024-08-19T01:31:35.594771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:46:37.551017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601362+0000","last_change":"2024-08-19T01:30:49.540136+0000","last_active":"2024-08-19T01:31:35.601362+0000","last_peered":"2024-08-19T01:31:35.601362+0000","last_clean":"2024-08-19T01:31:35.601362+0000","last_became_active":"2024-08-19T01:30:49.539680+0000","last_became_peered":"2024-08-19T01:30:49.539680+0000","last_unstale":"2024-08-19T01:31:35.601362+0000","last_undegraded":"2024-08-19T01:31:35.601362+0000","last_fullsized":"2024-08-19T01:31:35.601362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:25.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599759+0000","last_change":"2024-08-19T01:30:58.597130+0000","last_active":"2024-08-19T01:31:35.599759+0000","last_peered":"2024-08-19T01:31:35.599759+0000","last_clean":"2024-08-19T01:31:35.599759+0000","last_became_active":"2024-08-19T01:30:58.596834+0000","last_became_peered":"2024-08-19T01:30:58.596834+0000","last_unstale":"2024-08-19T01:31:35.599759+0000","last_undegraded":"2024-08-19T01:31:35.599759+0000","last_fullsized":"2024-08-19T01:31:35.599759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:13:10.441421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868410+0000","last_change":"2024-08-19T01:30:51.555575+0000","last_active":"2024-08-19T01:31:35.868410+0000","last_peered":"2024-08-19T01:31:35.868410+0000","last_clean":"2024-08-19T01:31:35.868410+0000","last_became_active":"2024-08-19T01:30:51.555309+0000","last_became_peered":"2024-08-19T01:30:51.555309+0000","last_unstale":"2024-08-19T01:31:35.868410+0000","last_undegraded":"2024-08-19T01:31:35.868410+0000","last_fullsized":"2024-08-19T01:31:35.868410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:11:25.515286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.744964+0000","last_change":"2024-08-19T01:30:49.536844+0000","last_active":"2024-08-19T01:31:35.744964+0000","last_peered":"2024-08-19T01:31:35.744964+0000","last_clean":"2024-08-19T01:31:35.744964+0000","last_became_active":"2024-08-19T01:30:49.536496+0000","last_became_peered":"2024-08-19T01:30:49.536496+0000","last_unstale":"2024-08-19T01:31:35.744964+0000","last_undegraded":"2024-08-19T01:31:35.744964+0000","last_fullsized":"2024-08-19T01:31:35.744964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:04.917188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598260+0000","last_change":"2024-08-19T01:30:49.539356+0000","last_active":"2024-08-19T01:31:35.598260+0000","last_peered":"2024-08-19T01:31:35.598260+0000","last_clean":"2024-08-19T01:31:35.598260+0000","last_became_active":"2024-08-19T01:30:49.538953+0000","last_became_peered":"2024-08-19T01:30:49.538953+0000","last_unstale":"2024-08-19T01:31:35.598260+0000","last_undegraded":"2024-08-19T01:31:35.598260+0000","last_fullsized":"2024-08-19T01:31:35.598260+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:13:17.845140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.870021+0000","last_change":"2024-08-19T01:31:35.870021+0000","last_active":"2024-08-19T01:31:35.870021+0000","last_peered":"2024-08-19T01:31:35.870021+0000","last_clean":"2024-08-19T01:31:35.870021+0000","last_became_active":"2024-08-19T01:31:35.869742+0000","last_became_peered":"2024-08-19T01:31:35.869742+0000","last_unstale":"2024-08-19T01:31:35.870021+0000","last_undegraded":"2024-08-19T01:31:35.870021+0000","last_fullsized":"2024-08-19T01:31:35.870021+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:51.848694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597559+0000","last_change":"2024-08-19T01:30:58.590385+0000","last_active":"2024-08-19T01:31:35.597559+0000","last_peered":"2024-08-19T01:31:35.597559+0000","last_clean":"2024-08-19T01:31:35.597559+0000","last_became_active":"2024-08-19T01:30:58.590004+0000","last_became_peered":"2024-08-19T01:30:58.590004+0000","last_unstale":"2024-08-19T01:31:35.597559+0000","last_undegraded":"2024-08-19T01:31:35.597559+0000","last_fullsized":"2024-08-19T01:31:35.597559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:13:01.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:32:26.675833+0000","last_change":"2024-08-19T01:31:35.614962+0000","last_active":"2024-08-19T01:32:26.675833+0000","last_peered":"2024-08-19T01:32:26.675833+0000","last_clean":"2024-08-19T01:32:26.675833+0000","last_became_active":"2024-08-19T01:31:35.605904+0000","last_became_peered":"2024-08-19T01:31:35.605904+0000","last_unstale":"2024-08-19T01:32:26.675833+0000","last_undegraded":"2024-08-19T01:32:26.675833+0000","last_fullsized":"2024-08-19T01:32:26.675833+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:26:36.102805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603484+0000","last_change":"2024-08-19T01:30:51.550448+0000","last_active":"2024-08-19T01:31:35.603484+0000","last_peered":"2024-08-19T01:31:35.603484+0000","last_clean":"2024-08-19T01:31:35.603484+0000","last_became_active":"2024-08-19T01:30:51.550074+0000","last_became_peered":"2024-08-19T01:30:51.550074+0000","last_unstale":"2024-08-19T01:31:35.603484+0000","last_undegraded":"2024-08-19T01:31:35.603484+0000","last_fullsized":"2024-08-19T01:31:35.603484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:33:13.016183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597794+0000","last_change":"2024-08-19T01:30:58.592077+0000","last_active":"2024-08-19T01:31:35.597794+0000","last_peered":"2024-08-19T01:31:35.597794+0000","last_clean":"2024-08-19T01:31:35.597794+0000","last_became_active":"2024-08-19T01:30:58.591651+0000","last_became_peered":"2024-08-19T01:30:58.591651+0000","last_unstale":"2024-08-19T01:31:35.597794+0000","last_undegraded":"2024-08-19T01:31:35.597794+0000","last_fullsized":"2024-08-19T01:31:35.597794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:09:24.980954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599538+0000","last_change":"2024-08-19T01:30:51.552772+0000","last_active":"2024-08-19T01:31:35.599538+0000","last_peered":"2024-08-19T01:31:35.599538+0000","last_clean":"2024-08-19T01:31:35.599538+0000","last_became_active":"2024-08-19T01:30:51.550904+0000","last_became_peered":"2024-08-19T01:30:51.550904+0000","last_unstale":"2024-08-19T01:31:35.599538+0000","last_undegraded":"2024-08-19T01:31:35.599538+0000","last_fullsized":"2024-08-19T01:31:35.599538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:21:50.082587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598023+0000","last_change":"2024-08-19T01:30:51.559756+0000","last_active":"2024-08-19T01:31:35.598023+0000","last_peered":"2024-08-19T01:31:35.598023+0000","last_clean":"2024-08-19T01:31:35.598023+0000","last_became_active":"2024-08-19T01:30:51.559370+0000","last_became_peered":"2024-08-19T01:30:51.559370+0000","last_unstale":"2024-08-19T01:31:35.598023+0000","last_undegraded":"2024-08-19T01:31:35.598023+0000","last_fullsized":"2024-08-19T01:31:35.598023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:27:30.525346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597386+0000","last_change":"2024-08-19T01:30:51.556577+0000","last_active":"2024-08-19T01:31:35.597386+0000","last_peered":"2024-08-19T01:31:35.597386+0000","last_clean":"2024-08-19T01:31:35.597386+0000","last_became_active":"2024-08-19T01:30:51.555929+0000","last_became_peered":"2024-08-19T01:30:51.555929+0000","last_unstale":"2024-08-19T01:31:35.597386+0000","last_undegraded":"2024-08-19T01:31:35.597386+0000","last_fullsized":"2024-08-19T01:31:35.597386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:46:31.660752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598037+0000","last_change":"2024-08-19T01:30:51.558626+0000","last_active":"2024-08-19T01:31:35.598037+0000","last_peered":"2024-08-19T01:31:35.598037+0000","last_clean":"2024-08-19T01:31:35.598037+0000","last_became_active":"2024-08-19T01:30:51.558188+0000","last_became_peered":"2024-08-19T01:30:51.558188+0000","last_unstale":"2024-08-19T01:31:35.598037+0000","last_undegraded":"2024-08-19T01:31:35.598037+0000","last_fullsized":"2024-08-19T01:31:35.598037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:43:44.777643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745491+0000","last_change":"2024-08-19T01:30:51.552870+0000","last_active":"2024-08-19T01:31:35.745491+0000","last_peered":"2024-08-19T01:31:35.745491+0000","last_clean":"2024-08-19T01:31:35.745491+0000","last_became_active":"2024-08-19T01:30:51.552514+0000","last_became_peered":"2024-08-19T01:30:51.552514+0000","last_unstale":"2024-08-19T01:31:35.745491+0000","last_undegraded":"2024-08-19T01:31:35.745491+0000","last_fullsized":"2024-08-19T01:31:35.745491+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:10:58.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600636+0000","last_change":"2024-08-19T01:30:51.554216+0000","last_active":"2024-08-19T01:31:35.600636+0000","last_peered":"2024-08-19T01:31:35.600636+0000","last_clean":"2024-08-19T01:31:35.600636+0000","last_became_active":"2024-08-19T01:30:51.552535+0000","last_became_peered":"2024-08-19T01:30:51.552535+0000","last_unstale":"2024-08-19T01:31:35.600636+0000","last_undegraded":"2024-08-19T01:31:35.600636+0000","last_fullsized":"2024-08-19T01:31:35.600636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:26:52.971897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598164+0000","last_change":"2024-08-19T01:30:51.561703+0000","last_active":"2024-08-19T01:31:35.598164+0000","last_peered":"2024-08-19T01:31:35.598164+0000","last_clean":"2024-08-19T01:31:35.598164+0000","last_became_active":"2024-08-19T01:30:51.561385+0000","last_became_peered":"2024-08-19T01:30:51.561385+0000","last_unstale":"2024-08-19T01:31:35.598164+0000","last_undegraded":"2024-08-19T01:31:35.598164+0000","last_fullsized":"2024-08-19T01:31:35.598164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:07:23.187023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868940+0000","last_change":"2024-08-19T01:30:51.564047+0000","last_active":"2024-08-19T01:31:35.868940+0000","last_peered":"2024-08-19T01:31:35.868940+0000","last_clean":"2024-08-19T01:31:35.868940+0000","last_became_active":"2024-08-19T01:30:51.563743+0000","last_became_peered":"2024-08-19T01:30:51.563743+0000","last_unstale":"2024-08-19T01:31:35.868940+0000","last_undegraded":"2024-08-19T01:31:35.868940+0000","last_fullsized":"2024-08-19T01:31:35.868940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:50:37.639496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.601317+0000","last_change":"2024-08-19T01:30:51.557058+0000","last_active":"2024-08-19T01:31:35.601317+0000","last_peered":"2024-08-19T01:31:35.601317+0000","last_clean":"2024-08-19T01:31:35.601317+0000","last_became_active":"2024-08-19T01:30:51.554611+0000","last_became_peered":"2024-08-19T01:30:51.554611+0000","last_unstale":"2024-08-19T01:31:35.601317+0000","last_undegraded":"2024-08-19T01:31:35.601317+0000","last_fullsized":"2024-08-19T01:31:35.601317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:56:25.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868614+0000","last_change":"2024-08-19T01:30:51.564119+0000","last_active":"2024-08-19T01:31:35.868614+0000","last_peered":"2024-08-19T01:31:35.868614+0000","last_clean":"2024-08-19T01:31:35.868614+0000","last_became_active":"2024-08-19T01:30:51.563840+0000","last_became_peered":"2024-08-19T01:30:51.563840+0000","last_unstale":"2024-08-19T01:31:35.868614+0000","last_undegraded":"2024-08-19T01:31:35.868614+0000","last_fullsized":"2024-08-19T01:31:35.868614+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:44:44.928760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.596119+0000","last_change":"2024-08-19T01:30:51.561639+0000","last_active":"2024-08-19T01:31:35.596119+0000","last_peered":"2024-08-19T01:31:35.596119+0000","last_clean":"2024-08-19T01:31:35.596119+0000","last_became_active":"2024-08-19T01:30:51.561207+0000","last_became_peered":"2024-08-19T01:30:51.561207+0000","last_unstale":"2024-08-19T01:31:35.596119+0000","last_undegraded":"2024-08-19T01:31:35.596119+0000","last_fullsized":"2024-08-19T01:31:35.596119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:12:20.076628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745343+0000","last_change":"2024-08-19T01:30:51.556700+0000","last_active":"2024-08-19T01:31:35.745343+0000","last_peered":"2024-08-19T01:31:35.745343+0000","last_clean":"2024-08-19T01:31:35.745343+0000","last_became_active":"2024-08-19T01:30:51.556407+0000","last_became_peered":"2024-08-19T01:30:51.556407+0000","last_unstale":"2024-08-19T01:31:35.745343+0000","last_undegraded":"2024-08-19T01:31:35.745343+0000","last_fullsized":"2024-08-19T01:31:35.745343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:03:16.664253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599693+0000","last_change":"2024-08-19T01:30:51.553764+0000","last_active":"2024-08-19T01:31:35.599693+0000","last_peered":"2024-08-19T01:31:35.599693+0000","last_clean":"2024-08-19T01:31:35.599693+0000","last_became_active":"2024-08-19T01:30:51.553442+0000","last_became_peered":"2024-08-19T01:30:51.553442+0000","last_unstale":"2024-08-19T01:31:35.599693+0000","last_undegraded":"2024-08-19T01:31:35.599693+0000","last_fullsized":"2024-08-19T01:31:35.599693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:36:44.476979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598004+0000","last_change":"2024-08-19T01:30:51.565528+0000","last_active":"2024-08-19T01:31:35.598004+0000","last_peered":"2024-08-19T01:31:35.598004+0000","last_clean":"2024-08-19T01:31:35.598004+0000","last_became_active":"2024-08-19T01:30:51.564649+0000","last_became_peered":"2024-08-19T01:30:51.564649+0000","last_unstale":"2024-08-19T01:31:35.598004+0000","last_undegraded":"2024-08-19T01:31:35.598004+0000","last_fullsized":"2024-08-19T01:31:35.598004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:04:12.868496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595505+0000","last_change":"2024-08-19T01:30:51.553425+0000","last_active":"2024-08-19T01:31:35.595505+0000","last_peered":"2024-08-19T01:31:35.595505+0000","last_clean":"2024-08-19T01:31:35.595505+0000","last_became_active":"2024-08-19T01:30:51.553123+0000","last_became_peered":"2024-08-19T01:30:51.553123+0000","last_unstale":"2024-08-19T01:31:35.595505+0000","last_undegraded":"2024-08-19T01:31:35.595505+0000","last_fullsized":"2024-08-19T01:31:35.595505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:07.969228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600659+0000","last_change":"2024-08-19T01:30:51.556820+0000","last_active":"2024-08-19T01:31:35.600659+0000","last_peered":"2024-08-19T01:31:35.600659+0000","last_clean":"2024-08-19T01:31:35.600659+0000","last_became_active":"2024-08-19T01:30:51.556532+0000","last_became_peered":"2024-08-19T01:30:51.556532+0000","last_unstale":"2024-08-19T01:31:35.600659+0000","last_undegraded":"2024-08-19T01:31:35.600659+0000","last_fullsized":"2024-08-19T01:31:35.600659+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:40:37.053193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.595799+0000","last_change":"2024-08-19T01:30:51.561610+0000","last_active":"2024-08-19T01:31:35.595799+0000","last_peered":"2024-08-19T01:31:35.595799+0000","last_clean":"2024-08-19T01:31:35.595799+0000","last_became_active":"2024-08-19T01:30:51.561270+0000","last_became_peered":"2024-08-19T01:30:51.561270+0000","last_unstale":"2024-08-19T01:31:35.595799+0000","last_undegraded":"2024-08-19T01:31:35.595799+0000","last_fullsized":"2024-08-19T01:31:35.595799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:15:15.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597839+0000","last_change":"2024-08-19T01:30:51.558128+0000","last_active":"2024-08-19T01:31:35.597839+0000","last_peered":"2024-08-19T01:31:35.597839+0000","last_clean":"2024-08-19T01:31:35.597839+0000","last_became_active":"2024-08-19T01:30:51.557785+0000","last_became_peered":"2024-08-19T01:30:51.557785+0000","last_unstale":"2024-08-19T01:31:35.597839+0000","last_undegraded":"2024-08-19T01:31:35.597839+0000","last_fullsized":"2024-08-19T01:31:35.597839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:14.849212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.597900+0000","last_change":"2024-08-19T01:30:51.553917+0000","last_active":"2024-08-19T01:31:35.597900+0000","last_peered":"2024-08-19T01:31:35.597900+0000","last_clean":"2024-08-19T01:31:35.597900+0000","last_became_active":"2024-08-19T01:30:51.553596+0000","last_became_peered":"2024-08-19T01:30:51.553596+0000","last_unstale":"2024-08-19T01:31:35.597900+0000","last_undegraded":"2024-08-19T01:31:35.597900+0000","last_fullsized":"2024-08-19T01:31:35.597900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:03:36.001486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.745336+0000","last_change":"2024-08-19T01:30:51.555201+0000","last_active":"2024-08-19T01:31:35.745336+0000","last_peered":"2024-08-19T01:31:35.745336+0000","last_clean":"2024-08-19T01:31:35.745336+0000","last_became_active":"2024-08-19T01:30:51.554160+0000","last_became_peered":"2024-08-19T01:30:51.554160+0000","last_unstale":"2024-08-19T01:31:35.745336+0000","last_undegraded":"2024-08-19T01:31:35.745336+0000","last_fullsized":"2024-08-19T01:31:35.745336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:58:20.586931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600619+0000","last_change":"2024-08-19T01:30:51.560517+0000","last_active":"2024-08-19T01:31:35.600619+0000","last_peered":"2024-08-19T01:31:35.600619+0000","last_clean":"2024-08-19T01:31:35.600619+0000","last_became_active":"2024-08-19T01:30:51.557454+0000","last_became_peered":"2024-08-19T01:30:51.557454+0000","last_unstale":"2024-08-19T01:31:35.600619+0000","last_undegraded":"2024-08-19T01:31:35.600619+0000","last_fullsized":"2024-08-19T01:31:35.600619+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:41:08.192165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.604873+0000","last_change":"2024-08-19T01:30:51.565961+0000","last_active":"2024-08-19T01:31:35.604873+0000","last_peered":"2024-08-19T01:31:35.604873+0000","last_clean":"2024-08-19T01:31:35.604873+0000","last_became_active":"2024-08-19T01:30:51.565491+0000","last_became_peered":"2024-08-19T01:30:51.565491+0000","last_unstale":"2024-08-19T01:31:35.604873+0000","last_undegraded":"2024-08-19T01:31:35.604873+0000","last_fullsized":"2024-08-19T01:31:35.604873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:53:06.263411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.599327+0000","last_change":"2024-08-19T01:30:51.551297+0000","last_active":"2024-08-19T01:31:35.599327+0000","last_peered":"2024-08-19T01:31:35.599327+0000","last_clean":"2024-08-19T01:31:35.599327+0000","last_became_active":"2024-08-19T01:30:51.550929+0000","last_became_peered":"2024-08-19T01:30:51.550929+0000","last_unstale":"2024-08-19T01:31:35.599327+0000","last_undegraded":"2024-08-19T01:31:35.599327+0000","last_fullsized":"2024-08-19T01:31:35.599327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:07:03.625360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603232+0000","last_change":"2024-08-19T01:31:35.602883+0000","last_active":"2024-08-19T01:31:35.603232+0000","last_peered":"2024-08-19T01:31:35.603232+0000","last_clean":"2024-08-19T01:31:35.603232+0000","last_became_active":"2024-08-19T01:31:35.602571+0000","last_became_peered":"2024-08-19T01:31:35.602571+0000","last_unstale":"2024-08-19T01:31:35.603232+0000","last_undegraded":"2024-08-19T01:31:35.603232+0000","last_fullsized":"2024-08-19T01:31:35.603232+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:08:13.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.868405+0000","last_change":"2024-08-19T01:30:51.552548+0000","last_active":"2024-08-19T01:31:35.868405+0000","last_peered":"2024-08-19T01:31:35.868405+0000","last_clean":"2024-08-19T01:31:35.868405+0000","last_became_active":"2024-08-19T01:30:51.552264+0000","last_became_peered":"2024-08-19T01:30:51.552264+0000","last_unstale":"2024-08-19T01:31:35.868405+0000","last_undegraded":"2024-08-19T01:31:35.868405+0000","last_fullsized":"2024-08-19T01:31:35.868405+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:17:53.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603728+0000","last_change":"2024-08-19T01:30:51.558686+0000","last_active":"2024-08-19T01:31:35.603728+0000","last_peered":"2024-08-19T01:31:35.603728+0000","last_clean":"2024-08-19T01:31:35.603728+0000","last_became_active":"2024-08-19T01:30:51.558190+0000","last_became_peered":"2024-08-19T01:30:51.558190+0000","last_unstale":"2024-08-19T01:31:35.603728+0000","last_undegraded":"2024-08-19T01:31:35.603728+0000","last_fullsized":"2024-08-19T01:31:35.603728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:52:33.056948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598221+0000","last_change":"2024-08-19T01:30:51.548868+0000","last_active":"2024-08-19T01:31:35.598221+0000","last_peered":"2024-08-19T01:31:35.598221+0000","last_clean":"2024-08-19T01:31:35.598221+0000","last_became_active":"2024-08-19T01:30:51.548347+0000","last_became_peered":"2024-08-19T01:30:51.548347+0000","last_unstale":"2024-08-19T01:31:35.598221+0000","last_undegraded":"2024-08-19T01:31:35.598221+0000","last_fullsized":"2024-08-19T01:31:35.598221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:39:22.049290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.603369+0000","last_change":"2024-08-19T01:30:51.554054+0000","last_active":"2024-08-19T01:31:35.603369+0000","last_peered":"2024-08-19T01:31:35.603369+0000","last_clean":"2024-08-19T01:31:35.603369+0000","last_became_active":"2024-08-19T01:30:51.553731+0000","last_became_peered":"2024-08-19T01:30:51.553731+0000","last_unstale":"2024-08-19T01:31:35.603369+0000","last_undegraded":"2024-08-19T01:31:35.603369+0000","last_fullsized":"2024-08-19T01:31:35.603369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:25:17.939024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600261+0000","last_change":"2024-08-19T01:30:51.558662+0000","last_active":"2024-08-19T01:31:35.600261+0000","last_peered":"2024-08-19T01:31:35.600261+0000","last_clean":"2024-08-19T01:31:35.600261+0000","last_became_active":"2024-08-19T01:30:51.558131+0000","last_became_peered":"2024-08-19T01:30:51.558131+0000","last_unstale":"2024-08-19T01:31:35.600261+0000","last_undegraded":"2024-08-19T01:31:35.600261+0000","last_fullsized":"2024-08-19T01:31:35.600261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:31:25.054005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598117+0000","last_change":"2024-08-19T01:30:51.565170+0000","last_active":"2024-08-19T01:31:35.598117+0000","last_peered":"2024-08-19T01:31:35.598117+0000","last_clean":"2024-08-19T01:31:35.598117+0000","last_became_active":"2024-08-19T01:30:51.564765+0000","last_became_peered":"2024-08-19T01:30:51.564765+0000","last_unstale":"2024-08-19T01:31:35.598117+0000","last_undegraded":"2024-08-19T01:31:35.598117+0000","last_fullsized":"2024-08-19T01:31:35.598117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:00:48.668865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.598106+0000","last_change":"2024-08-19T01:30:51.555300+0000","last_active":"2024-08-19T01:31:35.598106+0000","last_peered":"2024-08-19T01:31:35.598106+0000","last_clean":"2024-08-19T01:31:35.598106+0000","last_became_active":"2024-08-19T01:30:51.554977+0000","last_became_peered":"2024-08-19T01:30:51.554977+0000","last_unstale":"2024-08-19T01:31:35.598106+0000","last_undegraded":"2024-08-19T01:31:35.598106+0000","last_fullsized":"2024-08-19T01:31:35.598106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:18:58.961783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-08-19T01:31:35.600410+0000","last_change":"2024-08-19T01:30:51.557906+0000","last_active":"2024-08-19T01:31:35.600410+0000","last_peered":"2024-08-19T01:31:35.600410+0000","last_clean":"2024-08-19T01:31:35.600410+0000","last_became_active":"2024-08-19T01:30:51.557598+0000","last_became_peered":"2024-08-19T01:30:51.557598+0000","last_unstale":"2024-08-19T01:31:35.600410+0000","last_undegraded":"2024-08-19T01:31:35.600410+0000","last_fullsized":"2024-08-19T01:31:35.600410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:33:12.785910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66460,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":340,"num_read_kb":236,"num_write":168,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4463,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":199380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282761,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481824,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":513466,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743461,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32296,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712856,"statfs":{"total":95995035648,"available":95961964544,"internally_reserved":0,"allocated":1146880,"data_stored":1163194,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906986,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":517798,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168621,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":513622,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430259,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1109920,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593784,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":508878,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822717,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1105118,"data_compressed":59514,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986241,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":575988,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215172,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":509072,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":509516,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":571870,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":510266,"data_compressed":23555,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34322,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T01:32:31.555 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-19T01:32:31.555 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-19T01:32:31.555 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-19T01:32:31.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-19T01:32:31.826 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:31.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:31 smithi064 ceph-mon[18515]: pgmap v319: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:31.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:31 smithi123 ceph-mon[21829]: pgmap v319: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:31.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:31 smithi161 ceph-mon[21757]: pgmap v319: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 -- 172.21.15.64:0/955224901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e0100b50 msgr2=0x7f70e0100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/955224901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e0100b50 0x7f70e0100f30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009900 tx=0x7f70d402a480 comp rx=0 tx=0).stop 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 -- 172.21.15.64:0/955224901 shutdown_connections 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/955224901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70e0102c20 0x7f70e010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/955224901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70e01022a0 0x7f70e01026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/955224901 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e0100b50 0x7f70e0100f30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 -- 172.21.15.64:0/955224901 >> 172.21.15.64:0/955224901 conn(0x7f70e00fc500 msgr2=0x7f70e00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:32.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 -- 172.21.15.64:0/955224901 shutdown_connections 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.344+0000 7f70e6dd0640 1 -- 172.21.15.64:0/955224901 wait complete. 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 Processor -- start 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 -- start start 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70e0100b50 0x7f70e0196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e01022a0 0x7f70e01973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70e0102c20 0x7f70e0197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f70e0110cb0 con 0x7f70e01022a0 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70e0110b30 con 0x7f70e0100b50 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70e6dd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70e0110e30 con 0x7f70e0102c20 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e01022a0 0x7f70e01973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70dffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e01022a0 0x7f70e01973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45468/0 (socket says 172.21.15.64:45468) 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.345+0000 7f70dffff640 1 -- 172.21.15.64:0/3768283800 learned_addr learned my addr 172.21.15.64:0/3768283800 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:32.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70e536d640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70e0102c20 0x7f70e0197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70e4b6c640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70e0100b50 0x7f70e0196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70dffff640 1 -- 172.21.15.64:0/3768283800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70e0102c20 msgr2=0x7f70e0197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70dffff640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70e0102c20 0x7f70e0197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70dffff640 1 -- 172.21.15.64:0/3768283800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70e0100b50 msgr2=0x7f70e0196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70dffff640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70e0100b50 0x7f70e0196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70dffff640 1 -- 172.21.15.64:0/3768283800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e01a6710 con 0x7f70e01022a0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70e4b6c640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70e0100b50 0x7f70e0196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70dffff640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e01022a0 0x7f70e01973e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00d770 tx=0x7f70cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70ddffb640 1 -- 172.21.15.64:0/3768283800 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc017070 con 0x7f70e01022a0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70ddffb640 1 -- 172.21.15.64:0/3768283800 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70cc005500 con 0x7f70e01022a0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70ddffb640 1 -- 172.21.15.64:0/3768283800 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc004e00 con 0x7f70e01022a0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e01a6a00 con 0x7f70e01022a0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.346+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e01a6ef0 con 0x7f70e01022a0 2024-08-19T01:32:32.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.348+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e0100f30 con 0x7f70e01022a0 2024-08-19T01:32:32.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.349+0000 7f70ddffb640 1 -- 172.21.15.64:0/3768283800 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f70cc004fa0 con 0x7f70e01022a0 2024-08-19T01:32:32.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.349+0000 7f70ddffb640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f70d00a2c10 0x7f70d00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:32.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.349+0000 7f70e4b6c640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f70d00a2c10 0x7f70d00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:32.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.349+0000 7f70ddffb640 1 -- 172.21.15.64:0/3768283800 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f70cc0cf0c0 con 0x7f70e01022a0 2024-08-19T01:32:32.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.350+0000 7f70e4b6c640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f70d00a2c10 0x7f70d00a50d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70d4000c00 tx=0x7f70d4004100 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:32.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.353+0000 7f70ddffb640 1 -- 172.21.15.64:0/3768283800 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f70cc09bb00 con 0x7f70e01022a0 2024-08-19T01:32:32.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.568+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f70e0108dc0 con 0x7f70e01022a0 2024-08-19T01:32:32.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.569+0000 7f70ddffb640 1 -- 172.21.15.64:0/3768283800 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f70cc09b400 con 0x7f70e01022a0 2024-08-19T01:32:32.570 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:32.570 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f70d00a2c10 msgr2=0x7f70d00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f70d00a2c10 0x7f70d00a50d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70d4000c00 tx=0x7f70d4004100 comp rx=0 tx=0).stop 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e01022a0 msgr2=0x7f70e01973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e01022a0 0x7f70e01973e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00d770 tx=0x7f70cc00dc30 comp rx=0 tx=0).stop 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 shutdown_connections 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f70d00a2c10 0x7f70d00a50d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70e0102c20 0x7f70e0197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f70e01022a0 0x7f70e01973e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 --2- 172.21.15.64:0/3768283800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70e0100b50 0x7f70e0196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 >> 172.21.15.64:0/3768283800 conn(0x7f70e00fc500 msgr2=0x7f70e00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 shutdown_connections 2024-08-19T01:32:32.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:32.571+0000 7f70e6dd0640 1 -- 172.21.15.64:0/3768283800 wait complete. 2024-08-19T01:32:32.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:32 smithi123 ceph-mon[21829]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:32:32.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:32 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3768283800' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-19T01:32:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:32 smithi161 ceph-mon[21757]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:32:32.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:32 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3768283800' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-19T01:32:32.955 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:32 smithi064 ceph-mon[18515]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:32:32.955 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:32 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3768283800' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-19T01:32:33.309 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-19T01:32:33.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:32:33.569 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:33.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:33 smithi123 ceph-mon[21829]: pgmap v320: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:33.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:33 smithi161 ceph-mon[21757]: pgmap v320: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:33.966 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:33 smithi064 ceph-mon[18515]: pgmap v320: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:34.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 -- 172.21.15.64:0/2989310157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520102c20 msgr2=0x7fd52010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:34.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 --2- 172.21.15.64:0/2989310157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520102c20 0x7fd52010d770 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd518009ca0 tx=0x7fd51802a950 comp rx=0 tx=0).stop 2024-08-19T01:32:34.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 -- 172.21.15.64:0/2989310157 shutdown_connections 2024-08-19T01:32:34.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 --2- 172.21.15.64:0/2989310157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520102c20 0x7fd52010d770 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 --2- 172.21.15.64:0/2989310157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5201022a0 0x7fd5201026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 --2- 172.21.15.64:0/2989310157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd520100b50 0x7fd520100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 -- 172.21.15.64:0/2989310157 >> 172.21.15.64:0/2989310157 conn(0x7fd5200fc500 msgr2=0x7fd5200fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 -- 172.21.15.64:0/2989310157 shutdown_connections 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 -- 172.21.15.64:0/2989310157 wait complete. 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 Processor -- start 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.047+0000 7fd527931640 1 -- start start 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd527931640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520100b50 0x7fd5200fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd527931640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5201022a0 0x7fd5200fcef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd527931640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd520102c20 0x7fd5200fd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd527931640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd520110cb0 con 0x7fd520100b50 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd527931640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd520110b30 con 0x7fd520102c20 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd527931640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd520110e30 con 0x7fd5201022a0 2024-08-19T01:32:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520100b50 0x7fd5200fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520100b50 0x7fd5200fc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45486/0 (socket says 172.21.15.64:45486) 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 -- 172.21.15.64:0/823897531 learned_addr learned my addr 172.21.15.64:0/823897531 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd525ece640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd520102c20 0x7fd5200fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd524ecc640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5201022a0 0x7fd5200fcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 -- 172.21.15.64:0/823897531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5201022a0 msgr2=0x7fd5200fcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5201022a0 0x7fd5200fcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 -- 172.21.15.64:0/823897531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd520102c20 msgr2=0x7fd5200fd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd520102c20 0x7fd5200fd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 -- 172.21.15.64:0/823897531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5201a65c0 con 0x7fd520100b50 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd524ecc640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5201022a0 0x7fd5200fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5256cd640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520100b50 0x7fd5200fc9b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd51400d950 tx=0x7fd51400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5127fc640 1 -- 172.21.15.64:0/823897531 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514017030 con 0x7fd520100b50 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5127fc640 1 -- 172.21.15.64:0/823897531 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd514004020 con 0x7fd520100b50 2024-08-19T01:32:34.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5201a68b0 con 0x7fd520100b50 2024-08-19T01:32:34.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.048+0000 7fd5127fc640 1 -- 172.21.15.64:0/823897531 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5140047a0 con 0x7fd520100b50 2024-08-19T01:32:34.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.049+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5201a6e40 con 0x7fd520100b50 2024-08-19T01:32:34.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.049+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd520100f30 con 0x7fd520100b50 2024-08-19T01:32:34.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.051+0000 7fd5127fc640 1 -- 172.21.15.64:0/823897531 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd5140041d0 con 0x7fd520100b50 2024-08-19T01:32:34.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.051+0000 7fd5127fc640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd50c0a2c90 0x7fd50c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:34.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.051+0000 7fd524ecc640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd50c0a2c90 0x7fd50c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:34.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.051+0000 7fd5127fc640 1 -- 172.21.15.64:0/823897531 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd5140ceea0 con 0x7fd520100b50 2024-08-19T01:32:34.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.052+0000 7fd524ecc640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd50c0a2c90 0x7fd50c0a5150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd51c005e10 tx=0x7fd51c005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:34.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.055+0000 7fd5127fc640 1 -- 172.21.15.64:0/823897531 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd51409b940 con 0x7fd520100b50 2024-08-19T01:32:34.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.265+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5200fdee0 con 0x7fd520100b50 2024-08-19T01:32:34.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.266+0000 7fd5127fc640 1 -- 172.21.15.64:0/823897531 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11921 (secure 0 0 0) 0x7fd51409b240 con 0x7fd520100b50 2024-08-19T01:32:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:34.268 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"btime":"2024-08-19T01:31:35:584965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:35.353608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:32:34.268 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 24 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd50c0a2c90 msgr2=0x7fd50c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd50c0a2c90 0x7fd50c0a5150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd51c005e10 tx=0x7fd51c005da0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520100b50 msgr2=0x7fd5200fc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520100b50 0x7fd5200fc9b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd51400d950 tx=0x7fd51400de10 comp rx=0 tx=0).stop 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 shutdown_connections 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd50c0a2c90 0x7fd50c0a5150 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd520102c20 0x7fd5200fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5201022a0 0x7fd5200fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 --2- 172.21.15.64:0/823897531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd520100b50 0x7fd5200fc9b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:34.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 >> 172.21.15.64:0/823897531 conn(0x7fd5200fc500 msgr2=0x7fd520069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:34.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 shutdown_connections 2024-08-19T01:32:34.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:34.268+0000 7fd527931640 1 -- 172.21.15.64:0/823897531 wait complete. 2024-08-19T01:32:34.905 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:34 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/823897531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:34.905 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:34 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-08-19T01:32:34.905 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:34 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-08-19T01:32:34.905 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:34 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-08-19T01:32:34.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:34 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-08-19T01:32:34.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:34 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/823897531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:34.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:34 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-08-19T01:32:34.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:34 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-08-19T01:32:34.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:34 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-08-19T01:32:34.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:34 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-08-19T01:32:34.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:32:34.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:34 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/823897531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:34.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:34 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-08-19T01:32:34.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:34 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-08-19T01:32:34.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:34 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-08-19T01:32:34.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:34 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-08-19T01:32:35.186 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.657+0000 7fa09d0ff640 1 -- 172.21.15.64:0/1596477698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa098102280 msgr2=0x7fa09810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.657+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/1596477698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa098102280 0x7fa09810e7b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa090009cd0 tx=0x7fa09002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.657+0000 7fa09d0ff640 1 -- 172.21.15.64:0/1596477698 shutdown_connections 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.657+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/1596477698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa098102280 0x7fa09810e7b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.657+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/1596477698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0981018e0 0x7fa098101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.657+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/1596477698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0981078e0 0x7fa098107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.657+0000 7fa09d0ff640 1 -- 172.21.15.64:0/1596477698 >> 172.21.15.64:0/1596477698 conn(0x7fa0980fd600 msgr2=0x7fa0980ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.658+0000 7fa09d0ff640 1 -- 172.21.15.64:0/1596477698 shutdown_connections 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.658+0000 7fa09d0ff640 1 -- 172.21.15.64:0/1596477698 wait complete. 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.658+0000 7fa09d0ff640 1 Processor -- start 2024-08-19T01:32:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09d0ff640 1 -- start start 2024-08-19T01:32:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09d0ff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0981018e0 0x7fa098198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09d0ff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa098102280 0x7fa0981986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09d0ff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0981078e0 0x7fa09819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09d0ff640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa098111ce0 con 0x7fa0981018e0 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09d0ff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa098111b60 con 0x7fa098102280 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09d0ff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa098111e60 con 0x7fa0981078e0 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0981018e0 0x7fa098198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0981018e0 0x7fa098198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45502/0 (socket says 172.21.15.64:45502) 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 -- 172.21.15.64:0/2378679567 learned_addr learned my addr 172.21.15.64:0/2378679567 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09659c640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa098102280 0x7fa0981986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa09759e640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0981078e0 0x7fa09819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 -- 172.21.15.64:0/2378679567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0981078e0 msgr2=0x7fa09819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0981078e0 0x7fa09819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 -- 172.21.15.64:0/2378679567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa098102280 msgr2=0x7fa0981986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:35.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa098102280 0x7fa0981986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 -- 172.21.15.64:0/2378679567 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09819d130 con 0x7fa0981018e0 2024-08-19T01:32:35.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.659+0000 7fa096d9d640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0981018e0 0x7fa098198180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa08800d950 tx=0x7fa08800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:35.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.664+0000 7fa07ffff640 1 -- 172.21.15.64:0/2378679567 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa088017030 con 0x7fa0981018e0 2024-08-19T01:32:35.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.664+0000 7fa07ffff640 1 -- 172.21.15.64:0/2378679567 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa088004020 con 0x7fa0981018e0 2024-08-19T01:32:35.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.664+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa09819d420 con 0x7fa0981018e0 2024-08-19T01:32:35.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.664+0000 7fa07ffff640 1 -- 172.21.15.64:0/2378679567 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0880047a0 con 0x7fa0981018e0 2024-08-19T01:32:35.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.664+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0981a4d00 con 0x7fa0981018e0 2024-08-19T01:32:35.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.665+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa098101d40 con 0x7fa0981018e0 2024-08-19T01:32:35.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.666+0000 7fa07ffff640 1 -- 172.21.15.64:0/2378679567 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa0880041d0 con 0x7fa0981018e0 2024-08-19T01:32:35.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.667+0000 7fa07ffff640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa0800a2c90 0x7fa0800a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:35.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.667+0000 7fa07ffff640 1 -- 172.21.15.64:0/2378679567 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa0880cedf0 con 0x7fa0981018e0 2024-08-19T01:32:35.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.667+0000 7fa09659c640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa0800a2c90 0x7fa0800a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:35.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.668+0000 7fa09659c640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa0800a2c90 0x7fa0800a5150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa098106fd0 tx=0x7fa08c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:35.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.672+0000 7fa07ffff640 1 -- 172.21.15.64:0/2378679567 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa08809b820 con 0x7fa0981018e0 2024-08-19T01:32:35.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.885+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0981994e0 con 0x7fa0981018e0 2024-08-19T01:32:35.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:35 smithi064 ceph-mon[18515]: pgmap v321: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:35.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-08-19T01:32:35.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-08-19T01:32:35.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-08-19T01:32:35.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:35 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-08-19T01:32:35.887 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:35 smithi064 ceph-mon[18515]: osdmap e92: 12 total, 12 up, 12 in 2024-08-19T01:32:35.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.886+0000 7fa07ffff640 1 -- 172.21.15.64:0/2378679567 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11921 (secure 0 0 0) 0x7fa08809b120 con 0x7fa0981018e0 2024-08-19T01:32:35.888 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:35.888 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"btime":"2024-08-19T01:31:35:584965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:35.353608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:32:35.888 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 24 2024-08-19T01:32:35.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa0800a2c90 msgr2=0x7fa0800a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa0800a2c90 0x7fa0800a5150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa098106fd0 tx=0x7fa08c00b390 comp rx=0 tx=0).stop 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0981018e0 msgr2=0x7fa098198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0981018e0 0x7fa098198180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa08800d950 tx=0x7fa08800de10 comp rx=0 tx=0).stop 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 shutdown_connections 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fa0800a2c90 0x7fa0800a5150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0981078e0 0x7fa09819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa098102280 0x7fa0981986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 --2- 172.21.15.64:0/2378679567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0981018e0 0x7fa098198180 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 >> 172.21.15.64:0/2378679567 conn(0x7fa0980fd600 msgr2=0x7fa0980fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 shutdown_connections 2024-08-19T01:32:35.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:35.888+0000 7fa09d0ff640 1 -- 172.21.15.64:0/2378679567 wait complete. 2024-08-19T01:32:35.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:35 smithi123 ceph-mon[21829]: pgmap v321: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:35.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-08-19T01:32:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-08-19T01:32:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-08-19T01:32:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:35 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-08-19T01:32:35.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:35 smithi123 ceph-mon[21829]: osdmap e92: 12 total, 12 up, 12 in 2024-08-19T01:32:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:35 smithi161 ceph-mon[21757]: pgmap v321: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-08-19T01:32:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-08-19T01:32:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-08-19T01:32:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:35 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-08-19T01:32:35.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:35 smithi161 ceph-mon[21757]: osdmap e92: 12 total, 12 up, 12 in 2024-08-19T01:32:36.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:32:36.861 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:36.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:36 smithi161 ceph-mon[21757]: osdmap e93: 12 total, 12 up, 12 in 2024-08-19T01:32:36.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:36 smithi161 ceph-mon[21757]: pgmap v324: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:32:36.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:36 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2378679567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:36.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:36 smithi064 ceph-mon[18515]: osdmap e93: 12 total, 12 up, 12 in 2024-08-19T01:32:36.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:36 smithi064 ceph-mon[18515]: pgmap v324: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:32:36.983 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:36 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2378679567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:37.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:36 smithi123 ceph-mon[21829]: osdmap e93: 12 total, 12 up, 12 in 2024-08-19T01:32:37.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:36 smithi123 ceph-mon[21829]: pgmap v324: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:32:37.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:36 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2378679567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:37.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 -- 172.21.15.64:0/3572295798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c102b10 msgr2=0x7f9a1c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3572295798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c102b10 0x7f9a1c102ef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009ca0 tx=0x7f9a0c02a950 comp rx=0 tx=0).stop 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 -- 172.21.15.64:0/3572295798 shutdown_connections 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3572295798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c10beb0 0x7f9a1c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3572295798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a1c103430 0x7f9a1c10b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3572295798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c102b10 0x7f9a1c102ef0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 -- 172.21.15.64:0/3572295798 >> 172.21.15.64:0/3572295798 conn(0x7f9a1c0fc4d0 msgr2=0x7f9a1c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 -- 172.21.15.64:0/3572295798 shutdown_connections 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.340+0000 7f9a20d30640 1 -- 172.21.15.64:0/3572295798 wait complete. 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 Processor -- start 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 -- start start 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a1c102b10 0x7f9a1c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:37.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c103430 0x7f9a1c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c10beb0 0x7f9a1c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c111c60 con 0x7f9a1c103430 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c111ae0 con 0x7f9a1c10beb0 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a20d30640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c111de0 con 0x7f9a1c102b10 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c103430 0x7f9a1c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c103430 0x7f9a1c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45518/0 (socket says 172.21.15.64:45518) 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 -- 172.21.15.64:0/3281466564 learned_addr learned my addr 172.21.15.64:0/3281466564 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a1a59c640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a1c102b10 0x7f9a1c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a1ad9d640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c10beb0 0x7f9a1c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 -- 172.21.15.64:0/3281466564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a1c102b10 msgr2=0x7f9a1c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a1c102b10 0x7f9a1c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 -- 172.21.15.64:0/3281466564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c10beb0 msgr2=0x7f9a1c19c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:37.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c10beb0 0x7f9a1c19c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a19d9b640 1 -- 172.21.15.64:0/3281466564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1c19d0a0 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a1a59c640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a1c102b10 0x7f9a1c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.341+0000 7f9a1ad9d640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c10beb0 0x7f9a1c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.342+0000 7f9a19d9b640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c103430 0x7f9a1c198630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c920 tx=0x7f9a1000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.342+0000 7f9a0b7fe640 1 -- 172.21.15.64:0/3281466564 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10007c70 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.342+0000 7f9a0b7fe640 1 -- 172.21.15.64:0/3281466564 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a10007e10 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.342+0000 7f9a0b7fe640 1 -- 172.21.15.64:0/3281466564 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10018890 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.342+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1c19d390 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.342+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a1c1a4c70 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.343+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a1c102ef0 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.344+0000 7f9a0b7fe640 1 -- 172.21.15.64:0/3281466564 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9a10018a50 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.345+0000 7f9a0b7fe640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a040a2c10 0x7f9a040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.345+0000 7f9a0b7fe640 1 -- 172.21.15.64:0/3281466564 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9a100cfa40 con 0x7f9a1c103430 2024-08-19T01:32:37.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.345+0000 7f9a1a59c640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a040a2c10 0x7f9a040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:37.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.345+0000 7f9a1a59c640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a040a2c10 0x7f9a040a50d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009a60 tx=0x7f9a0c005c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:37.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.348+0000 7f9a0b7fe640 1 -- 172.21.15.64:0/3281466564 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f9a10014030 con 0x7f9a1c103430 2024-08-19T01:32:37.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.528+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a1c199450 con 0x7f9a1c103430 2024-08-19T01:32:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.529+0000 7f9a0b7fe640 1 -- 172.21.15.64:0/3281466564 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24830 (secure 0 0 0) 0x7f9a1c199450 con 0x7f9a1c103430 2024-08-19T01:32:37.531 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:37.531 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":93,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:32:35.645177+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a040a2c10 msgr2=0x7f9a040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a040a2c10 0x7f9a040a50d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009a60 tx=0x7f9a0c005c30 comp rx=0 tx=0).stop 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c103430 msgr2=0x7f9a1c198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c103430 0x7f9a1c198630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c920 tx=0x7f9a1000cde0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 shutdown_connections 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f9a040a2c10 0x7f9a040a50d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a1c10beb0 0x7f9a1c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a1c103430 0x7f9a1c198630 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 --2- 172.21.15.64:0/3281466564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a1c102b10 0x7f9a1c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 >> 172.21.15.64:0/3281466564 conn(0x7f9a1c0fc4d0 msgr2=0x7f9a1c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:37.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.532+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 shutdown_connections 2024-08-19T01:32:37.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:37.533+0000 7f9a20d30640 1 -- 172.21.15.64:0/3281466564 wait complete. 2024-08-19T01:32:37.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:37 smithi064 ceph-mon[18515]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T01:32:37.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:37 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3281466564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:32:37.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:37 smithi161 ceph-mon[21757]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T01:32:37.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:37 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3281466564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:32:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:37 smithi123 ceph-mon[21829]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T01:32:38.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:37 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3281466564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:32:38.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:32:38.516 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:38.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:38 smithi064 ceph-mon[18515]: pgmap v325: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:38.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:38 smithi161 ceph-mon[21757]: pgmap v325: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:39.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 -- 172.21.15.64:0/1418952306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f810be80 msgr2=0x7fd7f810e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1418952306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f810be80 0x7fd7f810e320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0009ca0 tx=0x7fd7f002a950 comp rx=0 tx=0).stop 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 -- 172.21.15.64:0/1418952306 shutdown_connections 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1418952306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f810be80 0x7fd7f810e320 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1418952306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7f8103370 0x7fd7f810b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1418952306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f8102a50 0x7fd7f8102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 -- 172.21.15.64:0/1418952306 >> 172.21.15.64:0/1418952306 conn(0x7fd7f80fc4b0 msgr2=0x7fd7f80fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 -- 172.21.15.64:0/1418952306 shutdown_connections 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.982+0000 7fd7fea09640 1 -- 172.21.15.64:0/1418952306 wait complete. 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 Processor -- start 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 -- start start 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f8102a50 0x7fd7f8198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7f8103370 0x7fd7f8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f810be80 0x7fd7f819ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8111cc0 con 0x7fd7f8102a50 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8111b40 con 0x7fd7f810be80 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fea09640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8111e40 con 0x7fd7f8103370 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f8102a50 0x7fd7f8198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f8102a50 0x7fd7f8198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35386/0 (socket says 172.21.15.64:35386) 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 -- 172.21.15.64:0/1912642494 learned_addr learned my addr 172.21.15.64:0/1912642494 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f77fe640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7f8103370 0x7fd7f8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 -- 172.21.15.64:0/1912642494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7f8103370 msgr2=0x7fd7f8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7fcfa6640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f810be80 0x7fd7f819ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7f8103370 0x7fd7f8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 -- 172.21.15.64:0/1912642494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f810be80 msgr2=0x7fd7f819ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f810be80 0x7fd7f819ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f7fff640 1 -- 172.21.15.64:0/1912642494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f819d0f0 con 0x7fd7f8102a50 2024-08-19T01:32:39.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.983+0000 7fd7f77fe640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7f8103370 0x7fd7f8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.984+0000 7fd7f7fff640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f8102a50 0x7fd7f8198140 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800b4a0 tx=0x7fd7e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.984+0000 7fd7f57fa640 1 -- 172.21.15.64:0/1912642494 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8014070 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.984+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f819d3e0 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.984+0000 7fd7f57fa640 1 -- 172.21.15.64:0/1912642494 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e8010af0 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.984+0000 7fd7f57fa640 1 -- 172.21.15.64:0/1912642494 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8017d10 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.984+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f81a4cc0 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.985+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f8102e30 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.986+0000 7fd7f57fa640 1 -- 172.21.15.64:0/1912642494 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd7e8017070 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.987+0000 7fd7f57fa640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7e00a2c10 0x7fd7e00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.987+0000 7fd7f77fe640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7e00a2c10 0x7fd7e00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.987+0000 7fd7f57fa640 1 -- 172.21.15.64:0/1912642494 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd7e80ce910 con 0x7fd7f8102a50 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.987+0000 7fd7f77fe640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7e00a2c10 0x7fd7e00a50d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80fea40 tx=0x7fd7ec005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:38.991+0000 7fd7f57fa640 1 -- 172.21.15.64:0/1912642494 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd7e809b450 con 0x7fd7f8102a50 2024-08-19T01:32:39.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:38 smithi123 ceph-mon[21829]: pgmap v325: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:39.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.202+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7f8106f20 con 0x7fd7f8102a50 2024-08-19T01:32:39.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.202+0000 7fd7f57fa640 1 -- 172.21.15.64:0/1912642494 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11921 (secure 0 0 0) 0x7fd7e809ad50 con 0x7fd7f8102a50 2024-08-19T01:32:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:39.204 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"btime":"2024-08-19T01:31:35:584965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:35.353608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:32:39.204 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 24 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.204+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7e00a2c10 msgr2=0x7fd7e00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.204+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7e00a2c10 0x7fd7e00a50d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80fea40 tx=0x7fd7ec005e30 comp rx=0 tx=0).stop 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.204+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f8102a50 msgr2=0x7fd7f8198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.204+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f8102a50 0x7fd7f8198140 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800b4a0 tx=0x7fd7e800b960 comp rx=0 tx=0).stop 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 shutdown_connections 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fd7e00a2c10 0x7fd7e00a50d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f810be80 0x7fd7f819ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7f8103370 0x7fd7f8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 --2- 172.21.15.64:0/1912642494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7f8102a50 0x7fd7f8198140 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:39.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 >> 172.21.15.64:0/1912642494 conn(0x7fd7f80fc4b0 msgr2=0x7fd7f80fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:39.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 shutdown_connections 2024-08-19T01:32:39.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:39.205+0000 7fd7fea09640 1 -- 172.21.15.64:0/1912642494 wait complete. 2024-08-19T01:32:39.876 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-08-19T01:30:52.282253+0000', 'modified': '2024-08-19T01:31:35.353608+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24476, 'mds_1': 24452, 'mds_2': 24436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24415': {'gid': 24415, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/3530643584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3530643584}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 3530643584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6837/1326350329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 1326350329}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1326350329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6835/2709386676', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 2709386676}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2709386676}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.161:6839/2198702470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2198702470}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2198702470}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.123:6837/3425782983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6836', 'nonce': 3425782983}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 3425782983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/2281975354', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 2281975354}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 2281975354}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24476, 'qdb_cluster': [24476, 24452, 24436]} 2024-08-19T01:32:39.876 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-08-19T01:32:39.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.i status 2024-08-19T01:32:39.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:39 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1912642494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:39.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:39 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1912642494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:40.131 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:39 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1912642494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:40.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.591+0000 7ff8fad4f640 1 -- 172.21.15.64:0/606942943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f4100b50 msgr2=0x7ff8f4100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.591+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/606942943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f4100b50 0x7ff8f4100f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009900 tx=0x7ff8e802a480 comp rx=0 tx=0).stop 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- 172.21.15.64:0/606942943 shutdown_connections 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/606942943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8f4102c20 0x7ff8f410d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/606942943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8f41022a0 0x7ff8f41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/606942943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f4100b50 0x7ff8f4100f30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- 172.21.15.64:0/606942943 >> 172.21.15.64:0/606942943 conn(0x7ff8f40fc500 msgr2=0x7ff8f40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- 172.21.15.64:0/606942943 shutdown_connections 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- 172.21.15.64:0/606942943 wait complete. 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 Processor -- start 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- start start 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8f4100b50 0x7ff8f41981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f41022a0 0x7ff8f4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8f4102c20 0x7ff8f419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f4110cd0 con 0x7ff8f41022a0 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f4110b50 con 0x7ff8f4100b50 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.592+0000 7ff8fad4f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f4110e50 con 0x7ff8f4102c20 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f41022a0 0x7ff8f4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f41022a0 0x7ff8f4198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35406/0 (socket says 172.21.15.64:35406) 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3099814201 learned_addr learned my addr 172.21.15.64:0/3099814201 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8f4102c20 0x7ff8f419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8f4100b50 0x7ff8f41981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3099814201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8f4102c20 msgr2=0x7ff8f419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8f4102c20 0x7ff8f419caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3099814201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8f4100b50 msgr2=0x7ff8f41981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8f4100b50 0x7ff8f41981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3099814201 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f419d180 con 0x7ff8f41022a0 2024-08-19T01:32:40.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8f4102c20 0x7ff8f419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8f4100b50 0x7ff8f41981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f41022a0 0x7ff8f4198710 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c920 tx=0x7ff8e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f1ffb640 1 -- 172.21.15.64:0/3099814201 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0007c70 con 0x7ff8f41022a0 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f1ffb640 1 -- 172.21.15.64:0/3099814201 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e0007e10 con 0x7ff8f41022a0 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8f1ffb640 1 -- 172.21.15.64:0/3099814201 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0018890 con 0x7ff8f41022a0 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.593+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f419d470 con 0x7ff8f41022a0 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.594+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f41a4d50 con 0x7ff8f41022a0 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.596+0000 7ff8f1ffb640 1 -- 172.21.15.64:0/3099814201 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff8e0018a70 con 0x7ff8f41022a0 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.596+0000 7ff8fd14d740 1 Processor -- start 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.596+0000 7ff8fd14d740 1 -- start start 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.596+0000 7ff8f1ffb640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e40a2c90 0x7ff8e40a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577630 0x558ae8577a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577f50 0x558ae85822d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae8584c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558ae84dc710 con 0x558ae8577630 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558ae8474770 con 0x558ae8577f50 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558ae84d86d0 con 0x558ae8582810 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e40a2c90 0x7ff8e40a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577630 0x558ae8577a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f1ffb640 1 -- 172.21.15.64:0/3099814201 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff8e00cf770 con 0x7ff8f41022a0 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577630 0x558ae8577a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35412/0 (socket says 172.21.15.64:35412) 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 -- 172.21.15.64:0/3285890156 learned_addr learned my addr 172.21.15.64:0/3285890156 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae8584c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577f50 0x558ae85822d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 -- 172.21.15.64:0/3285890156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 msgr2=0x558ae8584c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae8584c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 -- 172.21.15.64:0/3285890156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577f50 msgr2=0x558ae85822d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577f50 0x558ae85822d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 -- 172.21.15.64:0/3285890156 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae847b3e0 con 0x558ae8577630 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577f50 0x558ae85822d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e40a2c90 0x7ff8e40a5150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec005fd0 tx=0x7ff8ec009e60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.597+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577630 0x558ae8577a10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009230 tx=0x7ff8e8004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f933ad997ad0378 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8deffd640 1 -- 172.21.15.64:0/3285890156 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8035030 con 0x558ae8577630 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8deffd640 1 -- 172.21.15.64:0/3285890156 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e80046c0 con 0x558ae8577630 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8deffd640 1 -- 172.21.15.64:0/3285890156 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8005d50 con 0x558ae8577630 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3285890156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577630 msgr2=0x558ae8577a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577630 0x558ae8577a10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009230 tx=0x7ff8e8004310 comp rx=0 tx=0).stop 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3285890156 shutdown_connections 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae8584c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577f50 0x558ae85822d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3285890156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577630 0x558ae8577a10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3285890156 >> 172.21.15.64:0/3285890156 conn(0x558ae8573bc0 msgr2=0x558ae8573fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:40.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3285890156 shutdown_connections 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.598+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3285890156 wait complete. 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 -- start start 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 Processor -- start 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae857f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577f50 0x558ae857fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8582810 0x558ae85800d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558ae85853e0 con 0x558ae8582810 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558ae8497970 con 0x558ae8577630 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558ae8585560 con 0x558ae8577f50 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8fd14d740 10 client.0 _ll_register_callbacks cb 0x558ae81eb7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8f92ec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8582810 0x558ae85800d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8f92ec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8582810 0x558ae85800d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35424/0 (socket says 172.21.15.64:35424) 2024-08-19T01:32:40.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8f92ec640 1 -- 172.21.15.64:0/3672646832 learned_addr learned my addr 172.21.15.64:0/3672646832 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae857f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.599+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577f50 0x558ae857fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f92ec640 1 -- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577f50 msgr2=0x558ae857fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577f50 0x558ae857fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f92ec640 1 -- 172.21.15.64:0/3672646832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 msgr2=0x558ae857f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae857f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f92ec640 1 -- 172.21.15.64:0/3672646832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae824e2b0 con 0x558ae8582810 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577f50 0x558ae857fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae857f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8582810 0x558ae85800d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec009fa0 tx=0x7ff8ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f9f10cca9d243b server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8d3fff640 1 -- 172.21.15.64:0/3672646832 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec013e50 con 0x558ae8582810 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8d3fff640 1 -- 172.21.15.64:0/3672646832 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff8ec02a400 con 0x558ae8582810 2024-08-19T01:32:40.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e80046c0 con 0x558ae8582810 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8d3fff640 1 -- 172.21.15.64:0/3672646832 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8ec033050 con 0x558ae8582810 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8fd14d740 10 client.15078 fetch_fsmap learned FSMap version 24 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558ae8590fa0 con 0x558ae8582810 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8d3fff640 1 -- 172.21.15.64:0/3672646832 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e80046c0 con 0x558ae8582810 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8d3fff640 1 -- 172.21.15.64:0/3672646832 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7ff8ec00f040 con 0x558ae8582810 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8fd14d740 10 client.15078 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.600+0000 7ff8fd14d740 10 client.15078 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.601+0000 7ff8fd14d740 20 client.15078 populate_metadata read hostname 'smithi064' 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.601+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8593620 0x558ae8594a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.601+0000 7ff8fd14d740 4 client.15078 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.601+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 --> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ae84e9c80 con 0x558ae8593620 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.601+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8593620 0x558ae8594a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.601+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8593620 0x558ae8594a90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=257be2163a81461b server_cookie=317c039843b0c9a4 in_seq=0 out_seq=0 2024-08-19T01:32:40.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.601+0000 7ff8d3fff640 10 client.15078 ms_handle_connect on v2:172.21.15.161:6838/2198702470 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8d3fff640 1 -- 172.21.15.64:0/3672646832 <== mds.2 v2:172.21.15.161:6838/2198702470 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558ae84e9c80 con 0x558ae8593620 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8d3fff640 10 client.15078 handle_command_reply: tid=0 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 client.15078 shutdown 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8582810 msgr2=0x558ae85800d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8582810 0x558ae85800d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec009fa0 tx=0x7ff8ec004ab0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 shutdown_connections 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8593620 0x558ae8594a90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8582810 0x558ae85800d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577f50 0x558ae857fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8d3fff640 0 client.15078 ms_handle_reset on v2:172.21.15.161:6838/2198702470 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3672646832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae857f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.602+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 >> 172.21.15.64:0/3672646832 conn(0x558ae8573bc0 msgr2=0x558ae8572d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.603+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 shutdown_connections 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.603+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3672646832 wait complete. 2024-08-19T01:32:40.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.603+0000 7ff8fd14d740 20 client.15078 trim_cache size 0 max 16384 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 Processor -- start 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 -- start start 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577630 0x558ae857a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae857ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8582810 0x558ae858dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558ae8592dd0 con 0x558ae8577f50 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558ae85934a0 con 0x558ae8582810 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558ae8593b70 con 0x558ae8577630 2024-08-19T01:32:40.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae857ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae857ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35432/0 (socket says 172.21.15.64:35432) 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3248478793 learned_addr learned my addr 172.21.15.64:0/3248478793 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577630 0x558ae857a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8582810 0x558ae858dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3248478793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577630 msgr2=0x558ae857a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577630 0x558ae857a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3248478793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8582810 msgr2=0x558ae858dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8582810 0x558ae858dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.621+0000 7ff8f3fff640 1 -- 172.21.15.64:0/3248478793 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae8584f40 con 0x558ae8577f50 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae857ef30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0007590 tx=0x7ff8e006c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5417c8fafab9e669 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8de7fc640 1 -- 172.21.15.64:0/3248478793 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e009fe50 con 0x558ae8577f50 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8de7fc640 1 -- 172.21.15.64:0/3248478793 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e00a4710 con 0x558ae8577f50 2024-08-19T01:32:40.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8de7fc640 1 -- 172.21.15.64:0/3248478793 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00cf440 con 0x558ae8577f50 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3248478793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 msgr2=0x558ae857ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae857ef30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0007590 tx=0x7ff8e006c1d0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3248478793 shutdown_connections 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8582810 0x558ae858dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae857ef30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/3248478793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8577630 0x558ae857a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3248478793 >> 172.21.15.64:0/3248478793 conn(0x558ae8573bc0 msgr2=0x558ae858d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3248478793 shutdown_connections 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.622+0000 7ff8fd14d740 1 -- 172.21.15.64:0/3248478793 wait complete. 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 -- start start 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 Processor -- start 2024-08-19T01:32:40.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae859d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae859d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae85965d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558ae8594910 con 0x558ae8577f50 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558ae8594240 con 0x558ae8577630 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558ae8594fe0 con 0x558ae8582810 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8fd14d740 10 client.0 _ll_register_callbacks cb 0x7ff8ec015540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae859d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae859d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35438/0 (socket says 172.21.15.64:35438) 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8f3fff640 1 -- 172.21.15.64:0/1014566691 learned_addr learned my addr 172.21.15.64:0/1014566691 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae859d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.623+0000 7ff8f92ec640 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae85965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8f3fff640 1 -- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 msgr2=0x558ae85965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae85965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8f3fff640 1 -- 172.21.15.64:0/1014566691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 msgr2=0x558ae859d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae859d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8f3fff640 1 -- 172.21.15.64:0/1014566691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae8596e00 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8f3fff640 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae859d630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff8e009ba30 tx=0x7ff8e009ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68142c756896ad78 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8d3fff640 1 -- 172.21.15.64:0/1014566691 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0018d60 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8d3fff640 1 -- 172.21.15.64:0/1014566691 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff8e00ccfa0 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e00a4710 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8d3fff640 1 -- 172.21.15.64:0/1014566691 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8e000fb30 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8fd14d740 10 client.15084 fetch_fsmap learned FSMap version 24 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558ae859b8b0 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.624+0000 7ff8d3fff640 1 -- 172.21.15.64:0/1014566691 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e00a4710 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8d3fff640 1 -- 172.21.15.64:0/1014566691 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7ff8e00a4280 con 0x558ae8577f50 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8fd14d740 10 client.15084 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:32:40.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8fd14d740 10 client.15084 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8fd14d740 20 client.15084 populate_metadata read hostname 'smithi064' 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8602020 0x558ae8600d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8fd14d740 4 client.15084 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 --> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] -- command(tid 0: {"prefix": "status"}) -- 0x558ae84e4190 con 0x558ae8602020 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8602020 0x558ae8600d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8f8aeb640 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8602020 0x558ae8600d30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=600f534a1c0e9c2b server_cookie=4ca8b3edfcd82207 in_seq=0 out_seq=0 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.625+0000 7ff8d3fff640 10 client.15084 ms_handle_connect on v2:172.21.15.161:6838/2198702470 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8d3fff640 1 -- 172.21.15.64:0/1014566691 <== mds.2 v2:172.21.15.161:6838/2198702470 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558ae84e4190 con 0x558ae8602020 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8d3fff640 10 client.15084 handle_command_reply: tid=0 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 client.15084 shutdown 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 msgr2=0x558ae859d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae859d630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff8e009ba30 tx=0x7ff8e009ff90 comp rx=0 tx=0).stop 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 shutdown_connections 2024-08-19T01:32:40.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x558ae8602020 0x558ae8600d30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558ae8582810 0x558ae85965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae8577f50 0x558ae859d630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 --2- 172.21.15.64:0/1014566691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558ae8577630 0x558ae859d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8d3fff640 0 client.15084 ms_handle_reset on v2:172.21.15.161:6838/2198702470 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 >> 172.21.15.64:0/1014566691 conn(0x558ae8573bc0 msgr2=0x558ae858dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 shutdown_connections 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 1 -- 172.21.15.64:0/1014566691 wait complete. 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fd14d740 20 client.15084 trim_cache size 0 max 16384 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:32:40.632 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24436, 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 67.882838018000001, 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 24, 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 93, 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 90, 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 128.09041599299999 2024-08-19T01:32:40.633 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e40a2c90 msgr2=0x7ff8e40a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e40a2c90 0x7ff8e40a5150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec005fd0 tx=0x7ff8ec009e60 comp rx=0 tx=0).stop 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f41022a0 msgr2=0x7ff8f4198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.626+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f41022a0 0x7ff8f4198710 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c920 tx=0x7ff8e000cde0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 shutdown_connections 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ff8e40a2c90 0x7ff8e40a5150 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8f4102c20 0x7ff8f419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8f41022a0 0x7ff8f4198710 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 --2- 172.21.15.64:0/3099814201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8f4100b50 0x7ff8f41981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:40.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 >> 172.21.15.64:0/3099814201 conn(0x7ff8f40fc500 msgr2=0x7ff8f4069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:40.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 shutdown_connections 2024-08-19T01:32:40.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:40.627+0000 7ff8fad4f640 1 -- 172.21.15.64:0/3099814201 wait complete. 2024-08-19T01:32:40.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:40 smithi161 ceph-mon[21757]: pgmap v326: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-08-19T01:32:40.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:40 smithi064 ceph-mon[18515]: pgmap v326: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-08-19T01:32:41.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:40 smithi123 ceph-mon[21829]: pgmap v326: 129 pgs: 4 peering, 125 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-08-19T01:32:41.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.h status 2024-08-19T01:32:41.743 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:42.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:41 smithi064 ceph-mon[18515]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-08-19T01:32:42.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:41 smithi064 ceph-mon[18515]: Cluster is now healthy 2024-08-19T01:32:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:41 smithi123 ceph-mon[21829]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-08-19T01:32:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:41 smithi123 ceph-mon[21829]: Cluster is now healthy 2024-08-19T01:32:42.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:41 smithi161 ceph-mon[21757]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-08-19T01:32:42.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:41 smithi161 ceph-mon[21757]: Cluster is now healthy 2024-08-19T01:32:42.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 -- 172.21.15.64:0/1571304337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab8100b50 msgr2=0x7faab8100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 --2- 172.21.15.64:0/1571304337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab8100b50 0x7faab8100f30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009ca0 tx=0x7faaa802a950 comp rx=0 tx=0).stop 2024-08-19T01:32:42.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 -- 172.21.15.64:0/1571304337 shutdown_connections 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 --2- 172.21.15.64:0/1571304337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faab8102c20 0x7faab810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 --2- 172.21.15.64:0/1571304337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faab81022a0 0x7faab81026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 --2- 172.21.15.64:0/1571304337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab8100b50 0x7faab8100f30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 -- 172.21.15.64:0/1571304337 >> 172.21.15.64:0/1571304337 conn(0x7faab80fc500 msgr2=0x7faab80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 -- 172.21.15.64:0/1571304337 shutdown_connections 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 -- 172.21.15.64:0/1571304337 wait complete. 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 Processor -- start 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.193+0000 7faabda92640 1 -- start start 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faabda92640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faab8100b50 0x7faab81970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faabda92640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab81022a0 0x7faab8197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faabda92640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faab8102c20 0x7faab819b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faabda92640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faab8110cb0 con 0x7faab81022a0 2024-08-19T01:32:42.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faabda92640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faab8110b30 con 0x7faab8100b50 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faabda92640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faab8110e30 con 0x7faab8102c20 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab81022a0 0x7faab8197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab81022a0 0x7faab8197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35454/0 (socket says 172.21.15.64:35454) 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 -- 172.21.15.64:0/530135465 learned_addr learned my addr 172.21.15.64:0/530135465 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab77fe640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faab8100b50 0x7faab81970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab7fff640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faab8102c20 0x7faab819b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 -- 172.21.15.64:0/530135465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faab8102c20 msgr2=0x7faab819b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faab8102c20 0x7faab819b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 -- 172.21.15.64:0/530135465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faab8100b50 msgr2=0x7faab81970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faab8100b50 0x7faab81970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 -- 172.21.15.64:0/530135465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab819c0c0 con 0x7faab81022a0 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab6ffd640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab81022a0 0x7faab8197630 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faaa000d950 tx=0x7faaa000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab4ff9640 1 -- 172.21.15.64:0/530135465 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa0017030 con 0x7faab81022a0 2024-08-19T01:32:42.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.194+0000 7faab4ff9640 1 -- 172.21.15.64:0/530135465 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa0004020 con 0x7faab81022a0 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.195+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab819c3b0 con 0x7faab81022a0 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.195+0000 7faab4ff9640 1 -- 172.21.15.64:0/530135465 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa00047a0 con 0x7faab81022a0 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.195+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab8069ca0 con 0x7faab81022a0 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.197+0000 7faab4ff9640 1 -- 172.21.15.64:0/530135465 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7faaa00041d0 con 0x7faab81022a0 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.197+0000 7faabfe90740 1 Processor -- start 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.197+0000 7faabfe90740 1 -- start start 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.197+0000 7faabfe90740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea54a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faabfe90740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea5f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faabfe90740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea61c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faabfe90740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561efe9b9710 con 0x561efea54f50 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faabfe90740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561efe951770 con 0x561efea54630 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faabfe90740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561efe9b56d0 con 0x561efea5f810 2024-08-19T01:32:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea5f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea5f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35456/0 (socket says 172.21.15.64:35456) 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 -- 172.21.15.64:0/1826739380 learned_addr learned my addr 172.21.15.64:0/1826739380 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab6ffd640 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea61c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab7fff640 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea54a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 -- 172.21.15.64:0/1826739380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 msgr2=0x561efea61c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea61c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 -- 172.21.15.64:0/1826739380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 msgr2=0x561efea54a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea54a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab4ff9640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7faaa40a2c10 0x7faaa40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 -- 172.21.15.64:0/1826739380 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561efe9583e0 con 0x561efea54f50 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab6ffd640 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea61c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab7fff640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7faaa40a2c10 0x7faaa40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab77fe640 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea5f2d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faaa8004ad0 tx=0x7faaa80098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624d1da56077adc5 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faab4ff9640 1 -- 172.21.15.64:0/530135465 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faaa00cef60 con 0x7faab81022a0 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faaadffb640 1 -- 172.21.15.64:0/1826739380 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8048070 con 0x561efea54f50 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.198+0000 7faaadffb640 1 -- 172.21.15.64:0/1826739380 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa802f930 con 0x561efea54f50 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faaadffb640 1 -- 172.21.15.64:0/1826739380 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8034040 con 0x561efea54f50 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 -- 172.21.15.64:0/1826739380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 msgr2=0x561efea5f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea5f2d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faaa8004ad0 tx=0x7faaa80098d0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faab7fff640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7faaa40a2c10 0x7faaa40a50d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faab000b6c0 tx=0x7faab000b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 -- 172.21.15.64:0/1826739380 shutdown_connections 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea61c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea5f2d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 --2- 172.21.15.64:0/1826739380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea54a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 -- 172.21.15.64:0/1826739380 >> 172.21.15.64:0/1826739380 conn(0x561efea50bc0 msgr2=0x561efea50fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 -- 172.21.15.64:0/1826739380 shutdown_connections 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.199+0000 7faabfe90740 1 -- 172.21.15.64:0/1826739380 wait complete. 2024-08-19T01:32:42.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 -- start start 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 Processor -- start 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54630 0x561efea5c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54f50 0x561efea5cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea5d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561efea623e0 con 0x561efea54630 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561efe974970 con 0x561efea5f810 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561efea62560 con 0x561efea54f50 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faabfe90740 10 client.0 _ll_register_callbacks cb 0x561efe6c87b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54630 0x561efea5c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54630 0x561efea5c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35462/0 (socket says 172.21.15.64:35462) 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 -- 172.21.15.64:0/2748874839 learned_addr learned my addr 172.21.15.64:0/2748874839 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab6ffd640 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54f50 0x561efea5cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab7fff640 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea5d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 -- 172.21.15.64:0/2748874839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54f50 msgr2=0x561efea5cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54f50 0x561efea5cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 -- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 msgr2=0x561efea5d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea5d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.200+0000 7faab77fe640 1 -- 172.21.15.64:0/2748874839 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561efe72b2b0 con 0x561efea54630 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faab77fe640 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54630 0x561efea5c680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faaa8005e30 tx=0x7faaa802fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f2eff4b706e0fe server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faa96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faa96ffd640 1 -- 172.21.15.64:0/2748874839 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8048030 con 0x561efea54630 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faa96ffd640 1 -- 172.21.15.64:0/2748874839 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faaa8040050 con 0x561efea54630 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaa802f930 con 0x561efea54630 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faa96ffd640 1 -- 172.21.15.64:0/2748874839 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faaa8034040 con 0x561efea54630 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 10 client.15096 fetch_fsmap learned FSMap version 24 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561efea6df70 con 0x561efea54630 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faa96ffd640 1 -- 172.21.15.64:0/2748874839 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa802f930 con 0x561efea54630 2024-08-19T01:32:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faa96ffd640 1 -- 172.21.15.64:0/2748874839 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7faaa804e620 con 0x561efea54630 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 10 client.15096 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 10 client.15096 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 20 client.15096 populate_metadata read hostname 'smithi064' 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efea700c0 0x561efea71530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 4 client.15096 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.201+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 --> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561efe9c6c80 con 0x561efea700c0 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.202+0000 7faab6ffd640 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efea700c0 0x561efea71530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.202+0000 7faab6ffd640 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efea700c0 0x561efea71530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c8c84fd50ac47103 server_cookie=2d15c3742b4f8d13 in_seq=0 out_seq=0 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.202+0000 7faa96ffd640 10 client.15096 ms_handle_connect on v2:172.21.15.123:6836/3425782983 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faa96ffd640 1 -- 172.21.15.64:0/2748874839 <== mds.1 v2:172.21.15.123:6836/3425782983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561efe9c6c80 con 0x561efea700c0 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faa96ffd640 10 client.15096 handle_command_reply: tid=0 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 client.15096 shutdown 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54630 msgr2=0x561efea5c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54630 0x561efea5c680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faaa8005e30 tx=0x7faaa802fd10 comp rx=0 tx=0).stop 2024-08-19T01:32:42.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 shutdown_connections 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efea700c0 0x561efea71530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea5d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54f50 0x561efea5cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 --2- 172.21.15.64:0/2748874839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54630 0x561efea5c680 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faa96ffd640 0 client.15096 ms_handle_reset on v2:172.21.15.123:6836/3425782983 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 >> 172.21.15.64:0/2748874839 conn(0x561efea50bc0 msgr2=0x561efea568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 shutdown_connections 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.203+0000 7faabfe90740 1 -- 172.21.15.64:0/2748874839 wait complete. 2024-08-19T01:32:42.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.204+0000 7faabfe90740 20 client.15096 trim_cache size 0 max 16384 2024-08-19T01:32:42.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 Processor -- start 2024-08-19T01:32:42.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 -- start start 2024-08-19T01:32:42.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea6f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea6aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561efea6f870 con 0x561efea54f50 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561efea6ff40 con 0x561efea54630 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faabfe90740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561efea70610 con 0x561efea5f810 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea6aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea6aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35470/0 (socket says 172.21.15.64:35470) 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.222+0000 7faab6ffd640 1 -- 172.21.15.64:0/3952147026 learned_addr learned my addr 172.21.15.64:0/3952147026 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab77fe640 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea6f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab6ffd640 1 -- 172.21.15.64:0/3952147026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 msgr2=0x561efea598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab6ffd640 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab6ffd640 1 -- 172.21.15.64:0/3952147026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 msgr2=0x561efea6f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab6ffd640 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea6f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab6ffd640 1 -- 172.21.15.64:0/3952147026 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561efea54d80 con 0x561efea54f50 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab77fe640 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea6f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faab6ffd640 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea6aec0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faaa0005250 tx=0x7faaa00045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1390985a25c26dbc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faaad7fa640 1 -- 172.21.15.64:0/3952147026 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa009abf0 con 0x561efea54f50 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faaad7fa640 1 -- 172.21.15.64:0/3952147026 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa009b990 con 0x561efea54f50 2024-08-19T01:32:42.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.223+0000 7faaad7fa640 1 -- 172.21.15.64:0/3952147026 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa0099660 con 0x561efea54f50 2024-08-19T01:32:42.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 -- 172.21.15.64:0/3952147026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 msgr2=0x561efea6aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea6aec0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faaa0005250 tx=0x7faaa00045b0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 -- 172.21.15.64:0/3952147026 shutdown_connections 2024-08-19T01:32:42.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea5f810 0x561efea598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea6aec0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 --2- 172.21.15.64:0/3952147026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea54630 0x561efea6f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 -- 172.21.15.64:0/3952147026 >> 172.21.15.64:0/3952147026 conn(0x561efea50bc0 msgr2=0x561efea6a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 -- 172.21.15.64:0/3952147026 shutdown_connections 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 -- 172.21.15.64:0/3952147026 wait complete. 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 -- start start 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.224+0000 7faabfe90740 1 Processor -- start 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faabfe90740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54630 0x561efea75780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faabfe90740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea75cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faabfe90740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea76200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faabfe90740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561efea713b0 con 0x561efea54f50 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faabfe90740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561efea70ce0 con 0x561efea5f810 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faabfe90740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561efea71a80 con 0x561efea54630 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faabfe90740 10 client.0 _ll_register_callbacks cb 0x7faaa8046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:32:42.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea75cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea75cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35484/0 (socket says 172.21.15.64:35484) 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 -- 172.21.15.64:0/4027838574 learned_addr learned my addr 172.21.15.64:0/4027838574 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab7fff640 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea76200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab77fe640 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54630 0x561efea75780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 -- 172.21.15.64:0/4027838574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54630 msgr2=0x561efea75780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54630 0x561efea75780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 -- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 msgr2=0x561efea76200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea76200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 -- 172.21.15.64:0/4027838574 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561efead9a20 con 0x561efea54f50 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faab6ffd640 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea75cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faaa009af90 tx=0x7faaa00a3f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb757e148faed8a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faa96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.225+0000 7faa96ffd640 1 -- 172.21.15.64:0/4027838574 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa009f640 con 0x561efea54f50 2024-08-19T01:32:42.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faa96ffd640 1 -- 172.21.15.64:0/4027838574 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faaa00b4170 con 0x561efea54f50 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaa009b990 con 0x561efea54f50 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faa96ffd640 1 -- 172.21.15.64:0/4027838574 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faaa0097900 con 0x561efea54f50 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 10 client.15102 fetch_fsmap learned FSMap version 24 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561efeadac50 con 0x561efea54f50 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faa96ffd640 1 -- 172.21.15.64:0/4027838574 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa009b990 con 0x561efea54f50 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faa96ffd640 1 -- 172.21.15.64:0/4027838574 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7faaa009f120 con 0x561efea54f50 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 10 client.15102 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 10 client.15102 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 20 client.15102 populate_metadata read hostname 'smithi064' 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efeadf1f0 0x561efeaddf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 4 client.15102 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 --> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] -- command(tid 0: {"prefix": "status"}) -- 0x561efea5d640 con 0x561efeadf1f0 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.226+0000 7faab77fe640 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efeadf1f0 0x561efeaddf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faa96ffd640 10 client.15102 ms_handle_connect on v2:172.21.15.123:6836/3425782983 2024-08-19T01:32:42.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faab77fe640 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efeadf1f0 0x561efeaddf00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5bf46eaea988f24 server_cookie=51794ec1a05f229a in_seq=0 out_seq=0 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faa96ffd640 1 -- 172.21.15.64:0/4027838574 <== mds.1 v2:172.21.15.123:6836/3425782983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561efea5d640 con 0x561efeadf1f0 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faa96ffd640 10 client.15102 handle_command_reply: tid=0 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 client.15102 shutdown 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 msgr2=0x561efea75cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea75cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faaa009af90 tx=0x7faaa00a3f00 comp rx=0 tx=0).stop 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 shutdown_connections 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x561efeadf1f0 0x561efeaddf00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561efea5f810 0x561efea76200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561efea54f50 0x561efea75cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 --2- 172.21.15.64:0/4027838574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561efea54630 0x561efea75780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faa96ffd640 0 client.15102 ms_handle_reset on v2:172.21.15.123:6836/3425782983 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 >> 172.21.15.64:0/4027838574 conn(0x561efea50bc0 msgr2=0x561efea56810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 shutdown_connections 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 1 -- 172.21.15.64:0/4027838574 wait complete. 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.227+0000 7faabfe90740 20 client.15102 trim_cache size 0 max 16384 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:32:42.235 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24452, 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 70.489073945000001, 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 24, 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 93, 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 90, 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 132.31038873599999 2024-08-19T01:32:42.236 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:32:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7faaa40a2c10 msgr2=0x7faaa40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7faaa40a2c10 0x7faaa40a50d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faab000b6c0 tx=0x7faab000b2e0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab81022a0 msgr2=0x7faab8197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab81022a0 0x7faab8197630 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faaa000d950 tx=0x7faaa000de10 comp rx=0 tx=0).stop 2024-08-19T01:32:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 shutdown_connections 2024-08-19T01:32:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7faaa40a2c10 0x7faaa40a50d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faab8102c20 0x7faab819b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab81022a0 0x7faab8197630 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 --2- 172.21.15.64:0/530135465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faab8100b50 0x7faab81970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 >> 172.21.15.64:0/530135465 conn(0x7faab80fc500 msgr2=0x7faab80fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 shutdown_connections 2024-08-19T01:32:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:42.228+0000 7faabda92640 1 -- 172.21.15.64:0/530135465 wait complete. 2024-08-19T01:32:42.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:42 smithi064 ceph-mon[18515]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 14 B/s, 0 objects/s recovering 2024-08-19T01:32:43.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.b status 2024-08-19T01:32:43.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:42 smithi123 ceph-mon[21829]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 14 B/s, 0 objects/s recovering 2024-08-19T01:32:43.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:42 smithi161 ceph-mon[21757]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 14 B/s, 0 objects/s recovering 2024-08-19T01:32:43.321 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:43.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 -- 172.21.15.64:0/351586117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 msgr2=0x7f66d410d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 --2- 172.21.15.64:0/351586117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d410d770 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009900 tx=0x7f66cc02a480 comp rx=0 tx=0).stop 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 -- 172.21.15.64:0/351586117 shutdown_connections 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 --2- 172.21.15.64:0/351586117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d410d770 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 --2- 172.21.15.64:0/351586117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d41022a0 0x7f66d41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 --2- 172.21.15.64:0/351586117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66d4100b50 0x7f66d4100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 -- 172.21.15.64:0/351586117 >> 172.21.15.64:0/351586117 conn(0x7f66d40fc500 msgr2=0x7f66d40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 -- 172.21.15.64:0/351586117 shutdown_connections 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.781+0000 7f66d9183640 1 -- 172.21.15.64:0/351586117 wait complete. 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 Processor -- start 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 -- start start 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d4100b50 0x7f66d4198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66d41022a0 0x7f66d4199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d41a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f66d4110cd0 con 0x7f66d4102c20 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66d4110b50 con 0x7f66d41022a0 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d9183640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66d4110e50 con 0x7f66d4100b50 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d359e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d41a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d359e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d41a4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35496/0 (socket says 172.21.15.64:35496) 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d359e640 1 -- 172.21.15.64:0/150234269 learned_addr learned my addr 172.21.15.64:0/150234269 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d4100b50 0x7f66d4198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.782+0000 7f66d259c640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66d41022a0 0x7f66d4199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d359e640 1 -- 172.21.15.64:0/150234269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d4100b50 msgr2=0x7f66d4198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d359e640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d4100b50 0x7f66d4198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d359e640 1 -- 172.21.15.64:0/150234269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66d41022a0 msgr2=0x7f66d4199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d359e640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66d41022a0 0x7f66d4199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d359e640 1 -- 172.21.15.64:0/150234269 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d41a4590 con 0x7f66d4102c20 2024-08-19T01:32:43.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d359e640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d41a4050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f66cc005ab0 tx=0x7f66cc003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66cbfff640 1 -- 172.21.15.64:0/150234269 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc03f070 con 0x7f66d4102c20 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66cbfff640 1 -- 172.21.15.64:0/150234269 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66cc031e30 con 0x7f66d4102c20 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d41a4760 con 0x7f66d4102c20 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66cbfff640 1 -- 172.21.15.64:0/150234269 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc035030 con 0x7f66d4102c20 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.783+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d41a4bb0 con 0x7f66d4102c20 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66cbfff640 1 -- 172.21.15.64:0/150234269 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f66cc042070 con 0x7f66d4102c20 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 Processor -- start 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 -- start start 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31baa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 0x55adf31c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55adf311f710 con 0x55adf31ba630 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55adf30b7770 con 0x55adf31c5810 2024-08-19T01:32:43.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.786+0000 7f66db581740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55adf311b6d0 con 0x55adf31baf50 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31baa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31baa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35510/0 (socket says 172.21.15.64:35510) 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 -- 172.21.15.64:0/311948593 learned_addr learned my addr 172.21.15.64:0/311948593 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66cbfff640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f66c00a2c10 0x7f66c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 -- 172.21.15.64:0/311948593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 msgr2=0x55adf31c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 -- 172.21.15.64:0/311948593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 msgr2=0x55adf31c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 0x55adf31c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 -- 172.21.15.64:0/311948593 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf30be3e0 con 0x55adf31ba630 2024-08-19T01:32:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d359e640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f66c00a2c10 0x7f66c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66d259c640 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31baa10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00cca0 tx=0x7f66bc010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0710c9a1a5bc03 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66cbfff640 1 -- 172.21.15.64:0/150234269 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f66cc0efee0 con 0x7f66d4102c20 2024-08-19T01:32:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66c97fa640 1 -- 172.21.15.64:0/311948593 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc016070 con 0x55adf31ba630 2024-08-19T01:32:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66c97fa640 1 -- 172.21.15.64:0/311948593 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66bc0047a0 con 0x55adf31ba630 2024-08-19T01:32:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.787+0000 7f66c97fa640 1 -- 172.21.15.64:0/311948593 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc0128c0 con 0x55adf31ba630 2024-08-19T01:32:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 -- 172.21.15.64:0/311948593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 msgr2=0x55adf31baa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66d359e640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f66c00a2c10 0x7f66c00a50d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0b8670 tx=0x7f66cc0b16f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31baa10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00cca0 tx=0x7f66bc010740 comp rx=0 tx=0).stop 2024-08-19T01:32:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 -- 172.21.15.64:0/311948593 shutdown_connections 2024-08-19T01:32:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 0x55adf31c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 --2- 172.21.15.64:0/311948593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31baa10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 -- 172.21.15.64:0/311948593 >> 172.21.15.64:0/311948593 conn(0x55adf31b6bc0 msgr2=0x55adf31b6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 -- 172.21.15.64:0/311948593 shutdown_connections 2024-08-19T01:32:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.788+0000 7f66db581740 1 -- 172.21.15.64:0/311948593 wait complete. 2024-08-19T01:32:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 -- start start 2024-08-19T01:32:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 Processor -- start 2024-08-19T01:32:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 0x55adf31c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55adf31c83e0 con 0x55adf31ba630 2024-08-19T01:32:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55adf30da970 con 0x55adf31c5810 2024-08-19T01:32:43.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55adf31c8560 con 0x55adf31baf50 2024-08-19T01:32:43.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66db581740 10 client.0 _ll_register_callbacks cb 0x55adf2e2e7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:32:43.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35518/0 (socket says 172.21.15.64:35518) 2024-08-19T01:32:43.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 -- 172.21.15.64:0/690640163 learned_addr learned my addr 172.21.15.64:0/690640163 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 -- 172.21.15.64:0/690640163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 msgr2=0x55adf31c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 -- 172.21.15.64:0/690640163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 msgr2=0x55adf31c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 0x55adf31c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d259c640 1 -- 172.21.15.64:0/690640163 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf2e912b0 con 0x55adf31ba630 2024-08-19T01:32:43.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.789+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:43.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66d259c640 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31c2720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0124c0 tx=0x7f66bc004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae95a4d78a9ead10 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:43.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66ba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:32:43.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66ba7fc640 1 -- 172.21.15.64:0/690640163 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc016040 con 0x55adf31ba630 2024-08-19T01:32:43.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66ba7fc640 1 -- 172.21.15.64:0/690640163 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f66bc004030 con 0x55adf31ba630 2024-08-19T01:32:43.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66bc0047a0 con 0x55adf31ba630 2024-08-19T01:32:43.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66ba7fc640 1 -- 172.21.15.64:0/690640163 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f66bc02f050 con 0x55adf31ba630 2024-08-19T01:32:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 10 client.15114 fetch_fsmap learned FSMap version 24 2024-08-19T01:32:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55adf31d4010 con 0x55adf31ba630 2024-08-19T01:32:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66ba7fc640 1 -- 172.21.15.64:0/690640163 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66bc0047a0 con 0x55adf31ba630 2024-08-19T01:32:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66ba7fc640 1 -- 172.21.15.64:0/690640163 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f66bc0035f0 con 0x55adf31ba630 2024-08-19T01:32:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 10 client.15114 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:32:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 10 client.15114 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:32:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 20 client.15114 populate_metadata read hostname 'smithi064' 2024-08-19T01:32:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf31d6470 0x55adf31d78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 4 client.15114 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:32:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.790+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adf312cc80 con 0x55adf31d6470 2024-08-19T01:32:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.791+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf31d6470 0x55adf31d78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.791+0000 7f66ba7fc640 10 client.15114 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:32:43.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.791+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf31d6470 0x55adf31d78e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca931cd0d648a7b2 server_cookie=77ee287b099067d1 in_seq=0 out_seq=0 2024-08-19T01:32:43.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66ba7fc640 1 -- 172.21.15.64:0/690640163 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55adf312cc80 con 0x55adf31d6470 2024-08-19T01:32:43.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66ba7fc640 10 client.15114 handle_command_reply: tid=0 2024-08-19T01:32:43.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 client.15114 shutdown 2024-08-19T01:32:43.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 msgr2=0x55adf31c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31c2720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0124c0 tx=0x7f66bc004eb0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 shutdown_connections 2024-08-19T01:32:43.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf31d6470 0x55adf31d78e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31c5810 0x55adf31c31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31baf50 0x55adf31c2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66ba7fc640 0 client.15114 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:32:43.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 --2- 172.21.15.64:0/690640163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31c2720 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 >> 172.21.15.64:0/690640163 conn(0x55adf31b6bc0 msgr2=0x55adf31bc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:43.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 shutdown_connections 2024-08-19T01:32:43.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 1 -- 172.21.15.64:0/690640163 wait complete. 2024-08-19T01:32:43.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.792+0000 7f66db581740 20 client.15114 trim_cache size 0 max 16384 2024-08-19T01:32:43.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.810+0000 7f66db581740 1 Processor -- start 2024-08-19T01:32:43.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66db581740 1 -- start start 2024-08-19T01:32:43.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66db581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66db581740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 0x55adf31bd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66db581740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 0x55adf31c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66db581740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55adf31d5c20 con 0x55adf31ba630 2024-08-19T01:32:43.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66db581740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55adf31d62f0 con 0x55adf31baf50 2024-08-19T01:32:43.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66db581740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55adf31d69c0 con 0x55adf31c5810 2024-08-19T01:32:43.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35532/0 (socket says 172.21.15.64:35532) 2024-08-19T01:32:43.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 -- 172.21.15.64:0/2479555516 learned_addr learned my addr 172.21.15.64:0/2479555516 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:43.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 0x55adf31bd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 0x55adf31c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 -- 172.21.15.64:0/2479555516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 msgr2=0x55adf31c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 0x55adf31c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 -- 172.21.15.64:0/2479555516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 msgr2=0x55adf31bd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 0x55adf31bd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 -- 172.21.15.64:0/2479555516 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf31ba460 con 0x55adf31ba630 2024-08-19T01:32:43.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.811+0000 7f66d259c640 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d6b40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f66bc006010 tx=0x7f66bc003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98aacbf307beb562 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:43.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66c8ff9640 1 -- 172.21.15.64:0/2479555516 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc016020 con 0x55adf31ba630 2024-08-19T01:32:43.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66c8ff9640 1 -- 172.21.15.64:0/2479555516 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66bc023070 con 0x55adf31ba630 2024-08-19T01:32:43.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66c8ff9640 1 -- 172.21.15.64:0/2479555516 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc0035f0 con 0x55adf31ba630 2024-08-19T01:32:43.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66db581740 1 -- 172.21.15.64:0/2479555516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 msgr2=0x55adf31d6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66db581740 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d6b40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f66bc006010 tx=0x7f66bc003e00 comp rx=0 tx=0).stop 2024-08-19T01:32:43.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66db581740 1 -- 172.21.15.64:0/2479555516 shutdown_connections 2024-08-19T01:32:43.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66db581740 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 0x55adf31c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66db581740 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 0x55adf31bd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66db581740 1 --2- 172.21.15.64:0/2479555516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d6b40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.812+0000 7f66db581740 1 -- 172.21.15.64:0/2479555516 >> 172.21.15.64:0/2479555516 conn(0x55adf31b6bc0 msgr2=0x55adf31d5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:43.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 -- 172.21.15.64:0/2479555516 shutdown_connections 2024-08-19T01:32:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 -- 172.21.15.64:0/2479555516 wait complete. 2024-08-19T01:32:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 -- start start 2024-08-19T01:32:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 Processor -- start 2024-08-19T01:32:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 0x55adf31d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 0x55adf31df9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55adf31d7760 con 0x55adf31ba630 2024-08-19T01:32:43.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55adf31d7090 con 0x55adf31baf50 2024-08-19T01:32:43.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55adf31d7e30 con 0x55adf31c5810 2024-08-19T01:32:43.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66db581740 10 client.0 _ll_register_callbacks cb 0x7f66c40096d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:32:43.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66d259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35540/0 (socket says 172.21.15.64:35540) 2024-08-19T01:32:43.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66d259c640 1 -- 172.21.15.64:0/2303868420 learned_addr learned my addr 172.21.15.64:0/2303868420 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:43.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66d259c640 1 -- 172.21.15.64:0/2303868420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 msgr2=0x55adf31df9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66d259c640 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 0x55adf31df9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66d259c640 1 -- 172.21.15.64:0/2303868420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 msgr2=0x55adf31d98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:32:43.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.813+0000 7f66d259c640 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 0x55adf31d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66d259c640 1 -- 172.21.15.64:0/2303868420 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf31b2130 con 0x55adf31ba630 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66d259c640 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d93b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00c9a0 tx=0x7f66bc00c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c67796309202f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66ba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66ba7fc640 1 -- 172.21.15.64:0/2303868420 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc016070 con 0x55adf31ba630 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66ba7fc640 1 -- 172.21.15.64:0/2303868420 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f66bc004e20 con 0x55adf31ba630 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66bc023070 con 0x55adf31ba630 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66ba7fc640 1 -- 172.21.15.64:0/2303868420 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f66bc0035f0 con 0x55adf31ba630 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 10 client.15120 fetch_fsmap learned FSMap version 24 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55adf31ddf00 con 0x55adf31ba630 2024-08-19T01:32:43.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66ba7fc640 1 -- 172.21.15.64:0/2303868420 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66bc023070 con 0x55adf31ba630 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66ba7fc640 1 -- 172.21.15.64:0/2303868420 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f66bc012550 con 0x55adf31ba630 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 10 client.15120 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 20 client.15120 populate_metadata read hostname 'smithi064' 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf3244b00 0x55adf3243810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 4 client.15120 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.814+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "status"}) -- 0x55adf3127190 con 0x55adf3244b00 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf3244b00 0x55adf3243810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66d2d9d640 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf3244b00 0x55adf3243810 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=841e3ee3aaeeeb53 server_cookie=8869db2d19c3b97a in_seq=0 out_seq=0 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66ba7fc640 10 client.15120 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:32:43.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66ba7fc640 1 -- 172.21.15.64:0/2303868420 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55adf3127190 con 0x55adf3244b00 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66ba7fc640 10 client.15120 handle_command_reply: tid=0 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 client.15120 shutdown 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 msgr2=0x55adf31d93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d93b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f66bc00c9a0 tx=0x7f66bc00c9d0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 shutdown_connections 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55adf31c5810 0x55adf31df9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55adf3244b00 0x55adf3243810 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66ba7fc640 0 client.15120 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55adf31baf50 0x55adf31d98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.815+0000 7f66db581740 1 --2- 172.21.15.64:0/2303868420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55adf31ba630 0x55adf31d93b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 >> 172.21.15.64:0/2303868420 conn(0x55adf31b6bc0 msgr2=0x55adf31d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 shutdown_connections 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66db581740 1 -- 172.21.15.64:0/2303868420 wait complete. 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66db581740 20 client.15120 trim_cache size 0 max 16384 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2024-08-19T01:32:43.849 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24476, 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 111.26963023, 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 24, 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 93, 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 90, 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 122.27669351199999 2024-08-19T01:32:43.850 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:32:43.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f66c00a2c10 msgr2=0x7f66c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f66c00a2c10 0x7f66c00a50d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0b8670 tx=0x7f66cc0b16f0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 msgr2=0x7f66d41a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d41a4050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f66cc005ab0 tx=0x7f66cc003c30 comp rx=0 tx=0).stop 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 shutdown_connections 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f66c00a2c10 0x7f66c00a50d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f66d4102c20 0x7f66d41a4050 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66d41022a0 0x7f66d4199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 --2- 172.21.15.64:0/150234269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d4100b50 0x7f66d4198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.816+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 >> 172.21.15.64:0/150234269 conn(0x7f66d40fc500 msgr2=0x7f66d4106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.817+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 shutdown_connections 2024-08-19T01:32:43.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:43.817+0000 7f66d9183640 1 -- 172.21.15.64:0/150234269 wait complete. 2024-08-19T01:32:44.634 INFO:tasks.cephadm:Setup complete, yielding 2024-08-19T01:32:44.634 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-08-19T01:32:44.649 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:32:44.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- bash -c 'ceph orch status' 2024-08-19T01:32:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:45.162 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:44 smithi064 ceph-mon[18515]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 0 objects/s recovering 2024-08-19T01:32:45.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:44 smithi123 ceph-mon[21829]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 0 objects/s recovering 2024-08-19T01:32:45.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:44 smithi161 ceph-mon[21757]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 0 objects/s recovering 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.363+0000 7ffbb6248640 1 -- 172.21.15.64:0/1396987820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 msgr2=0x7ffbb0108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.363+0000 7ffbb6248640 1 --2- 172.21.15.64:0/1396987820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb0108c50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009ca0 tx=0x7ffba802a8c0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.363+0000 7ffbb6248640 1 -- 172.21.15.64:0/1396987820 shutdown_connections 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.363+0000 7ffbb6248640 1 --2- 172.21.15.64:0/1396987820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb0108c50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.363+0000 7ffbb6248640 1 --2- 172.21.15.64:0/1396987820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb0101a00 0x7ffbb0101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.363+0000 7ffbb6248640 1 --2- 172.21.15.64:0/1396987820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbb0100800 0x7ffbb0100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.363+0000 7ffbb6248640 1 -- 172.21.15.64:0/1396987820 >> 172.21.15.64:0/1396987820 conn(0x7ffbb00fbfb0 msgr2=0x7ffbb00fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 -- 172.21.15.64:0/1396987820 shutdown_connections 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 -- 172.21.15.64:0/1396987820 wait complete. 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 Processor -- start 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 -- start start 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbb0100800 0x7ffbb0197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb0101a00 0x7ffbb0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb019f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb010d680 con 0x7ffbb01023c0 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb010d500 con 0x7ffbb0100800 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.364+0000 7ffbb6248640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb010d800 con 0x7ffbb0101a00 2024-08-19T01:32:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb019f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb019f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35558/0 (socket says 172.21.15.64:35558) 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 -- 172.21.15.64:0/3316573376 learned_addr learned my addr 172.21.15.64:0/3316573376 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaeffd640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb0101a00 0x7ffbb0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaf7fe640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbb0100800 0x7ffbb0197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 -- 172.21.15.64:0/3316573376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb0101a00 msgr2=0x7ffbb0198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb0101a00 0x7ffbb0198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 -- 172.21.15.64:0/3316573376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbb0100800 msgr2=0x7ffbb0197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbb0100800 0x7ffbb0197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 -- 172.21.15.64:0/3316573376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb019f960 con 0x7ffbb01023c0 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbaffff640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb019f2f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ffba80098d0 tx=0x7ffba802f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbacff9640 1 -- 172.21.15.64:0/3316573376 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba802fc30 con 0x7ffbb01023c0 2024-08-19T01:32:45.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb019fbf0 con 0x7ffbb01023c0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbacff9640 1 -- 172.21.15.64:0/3316573376 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba8003e50 con 0x7ffbb01023c0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbacff9640 1 -- 172.21.15.64:0/3316573376 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba80305c0 con 0x7ffbb01023c0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.365+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb01a0180 con 0x7ffbb01023c0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.366+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb0107c70 con 0x7ffbb01023c0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.368+0000 7ffbacff9640 1 -- 172.21.15.64:0/3316573376 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ffba8004b40 con 0x7ffbb01023c0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.368+0000 7ffbacff9640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffb980a2c40 0x7ffb980a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.369+0000 7ffbaf7fe640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffb980a2c40 0x7ffb980a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.369+0000 7ffbacff9640 1 -- 172.21.15.64:0/3316573376 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffba80f0a30 con 0x7ffbb01023c0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.369+0000 7ffbaf7fe640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffb980a2c40 0x7ffb980a5100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0101860 tx=0x7ffba0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:45.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.372+0000 7ffbacff9640 1 -- 172.21.15.64:0/3316573376 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ffba80433d0 con 0x7ffbb01023c0 2024-08-19T01:32:45.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.563+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7ffbb0062360 con 0x7ffb980a2c40 2024-08-19T01:32:45.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.564+0000 7ffbacff9640 1 -- 172.21.15.64:0/3316573376 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7ffbb0062360 con 0x7ffb980a2c40 2024-08-19T01:32:45.565 INFO:teuthology.orchestra.run.smithi064.stdout:Backend: cephadm 2024-08-19T01:32:45.566 INFO:teuthology.orchestra.run.smithi064.stdout:Available: Yes 2024-08-19T01:32:45.566 INFO:teuthology.orchestra.run.smithi064.stdout:Paused: No 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffb980a2c40 msgr2=0x7ffb980a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffb980a2c40 0x7ffb980a5100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0101860 tx=0x7ffba0008040 comp rx=0 tx=0).stop 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 msgr2=0x7ffbb019f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb019f2f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ffba80098d0 tx=0x7ffba802f840 comp rx=0 tx=0).stop 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 shutdown_connections 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffb980a2c40 0x7ffb980a5100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbb01023c0 0x7ffbb019f2f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb0101a00 0x7ffbb0198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 --2- 172.21.15.64:0/3316573376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbb0100800 0x7ffbb0197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 >> 172.21.15.64:0/3316573376 conn(0x7ffbb00fbfb0 msgr2=0x7ffbb0104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 shutdown_connections 2024-08-19T01:32:45.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:45.566+0000 7ffbb6248640 1 -- 172.21.15.64:0/3316573376 wait complete. 2024-08-19T01:32:45.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:45 smithi064 ceph-mon[18515]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:46.067 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:45 smithi123 ceph-mon[21829]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:46.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:45 smithi161 ceph-mon[21757]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:46.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- bash -c 'ceph orch ps' 2024-08-19T01:32:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:46.976 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:46 smithi064 ceph-mon[18515]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 11 B/s, 0 objects/s recovering 2024-08-19T01:32:47.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 -- 172.21.15.64:0/3583473131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 msgr2=0x7f3438100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:47.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 --2- 172.21.15.64:0/3583473131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438100c40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3428009900 tx=0x7f342802a410 comp rx=0 tx=0).stop 2024-08-19T01:32:47.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 -- 172.21.15.64:0/3583473131 shutdown_connections 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 --2- 172.21.15.64:0/3583473131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3438102400 0x7f3438108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 --2- 172.21.15.64:0/3583473131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3438101a40 0x7f3438101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 --2- 172.21.15.64:0/3583473131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438100c40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 -- 172.21.15.64:0/3583473131 >> 172.21.15.64:0/3583473131 conn(0x7f34380fbfd0 msgr2=0x7f34380fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 -- 172.21.15.64:0/3583473131 shutdown_connections 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 -- 172.21.15.64:0/3583473131 wait complete. 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.057+0000 7f343cc52640 1 Processor -- start 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343cc52640 1 -- start start 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343cc52640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343cc52640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3438101a40 0x7f3438198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343cc52640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3438102400 0x7f343819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:47.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343cc52640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f343810d5f0 con 0x7f3438100840 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343cc52640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f343810d470 con 0x7f3438102400 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343cc52640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f343810d770 con 0x7f3438101a40 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35572/0 (socket says 172.21.15.64:35572) 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 -- 172.21.15.64:0/3748013231 learned_addr learned my addr 172.21.15.64:0/3748013231 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f3435d9b640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3438101a40 0x7f3438198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f3436d9d640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3438102400 0x7f343819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 -- 172.21.15.64:0/3748013231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3438101a40 msgr2=0x7f3438198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3438101a40 0x7f3438198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 -- 172.21.15.64:0/3748013231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3438102400 msgr2=0x7f343819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3438102400 0x7f343819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 -- 172.21.15.64:0/3748013231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343819f800 con 0x7f3438100840 2024-08-19T01:32:47.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f3436d9d640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3438102400 0x7f343819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f343659c640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438197bd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3428031610 tx=0x7f34280043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.058+0000 7f342f7fe640 1 -- 172.21.15.64:0/3748013231 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342803f030 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.059+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343819fa90 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.059+0000 7f342f7fe640 1 -- 172.21.15.64:0/3748013231 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3428003ce0 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.059+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343819ff70 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.059+0000 7f342f7fe640 1 -- 172.21.15.64:0/3748013231 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428049060 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.060+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3438107c40 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.061+0000 7f342f7fe640 1 -- 172.21.15.64:0/3748013231 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3428042030 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.061+0000 7f342f7fe640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34240a2c40 0x7f34240a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.062+0000 7f3435d9b640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34240a2c40 0x7f34240a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.062+0000 7f342f7fe640 1 -- 172.21.15.64:0/3748013231 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f34280efe90 con 0x7f3438100840 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.062+0000 7f3435d9b640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34240a2c40 0x7f34240a5100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f34381990f0 tx=0x7f342000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:47.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.066+0000 7f342f7fe640 1 -- 172.21.15.64:0/3748013231 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f34280bc8c0 con 0x7f3438100840 2024-08-19T01:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:46 smithi123 ceph-mon[21829]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 11 B/s, 0 objects/s recovering 2024-08-19T01:32:47.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:46 smithi161 ceph-mon[21757]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 11 B/s, 0 objects/s recovering 2024-08-19T01:32:47.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.249+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f3438062360 con 0x7f34240a2c40 2024-08-19T01:32:47.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.261+0000 7f342f7fe640 1 -- 172.21.15.64:0/3748013231 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f3438062360 con 0x7f34240a2c40 2024-08-19T01:32:47.262 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-19T01:32:47.262 INFO:teuthology.orchestra.run.smithi064.stdout:mds.a smithi064 running (2m) 57s ago 2m 16.6M - 19.1.0-1382-gde09dd01 ea7a5189f30b dd537a28c24f 2024-08-19T01:32:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:mds.b smithi123 running (2m) 57s ago 2m 26.5M - 19.1.0-1382-gde09dd01 ea7a5189f30b 532c3f8b51c4 2024-08-19T01:32:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:mds.c smithi161 running (2m) 57s ago 2m 15.7M - 19.1.0-1382-gde09dd01 ea7a5189f30b 1e98d414f9a4 2024-08-19T01:32:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:mds.d smithi064 running (2m) 57s ago 2m 15.8M - 19.1.0-1382-gde09dd01 ea7a5189f30b 41b5957e387d 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.e smithi123 running (2m) 57s ago 2m 16.6M - 19.1.0-1382-gde09dd01 ea7a5189f30b afb9d2fa33f8 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.f smithi161 running (2m) 57s ago 2m 17.2M - 19.1.0-1382-gde09dd01 ea7a5189f30b 3a9cfe841677 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.g smithi064 running (2m) 57s ago 2m 14.4M - 19.1.0-1382-gde09dd01 ea7a5189f30b f5862aed040d 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.h smithi123 running (2m) 57s ago 2m 16.5M - 19.1.0-1382-gde09dd01 ea7a5189f30b 7b76ed2d44d7 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.i smithi161 running (2m) 57s ago 2m 17.5M - 19.1.0-1382-gde09dd01 ea7a5189f30b 8a581b31383a 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.j smithi064 running (2m) 57s ago 2m 14.5M - 19.1.0-1382-gde09dd01 ea7a5189f30b 0b480a3d658e 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.k smithi123 running (2m) 57s ago 2m 14.5M - 19.1.0-1382-gde09dd01 ea7a5189f30b 0bd66baa8cf2 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mds.l smithi161 running (2m) 57s ago 2m 15.8M - 19.1.0-1382-gde09dd01 ea7a5189f30b 3427f5f088bb 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi064 *:9283,8765 running (8m) 57s ago 8m 508M - 19.1.0-1382-gde09dd01 ea7a5189f30b ce16acf65597 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi123 *:8443,8765 running (7m) 57s ago 7m 442M - 19.1.0-1382-gde09dd01 ea7a5189f30b 6c99f9ff5c16 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.z smithi161 *:8443,8765 running (7m) 57s ago 7m 441M - 19.1.0-1382-gde09dd01 ea7a5189f30b feace770023a 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (8m) 57s ago 8m 53.2M 2048M 19.1.0-1382-gde09dd01 ea7a5189f30b de0cedb2b02e 2024-08-19T01:32:47.266 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi123 running (7m) 57s ago 7m 41.5M 2048M 19.1.0-1382-gde09dd01 ea7a5189f30b b7d3a7dc2949 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi161 running (7m) 57s ago 7m 43.7M 2048M 19.1.0-1382-gde09dd01 ea7a5189f30b 425a47cf752d 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (7m) 57s ago 6m 45.7M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b f9d445611315 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi123 running (6m) 57s ago 6m 47.8M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b 6ef4f404cfc1 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi161 running (6m) 57s ago 6m 58.5M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b d3b317e57230 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (5m) 57s ago 5m 46.3M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b ccd8972e4529 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi123 running (5m) 57s ago 5m 58.0M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b 05f86b31fc86 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi161 running (5m) 57s ago 5m 45.2M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b 874064f2c127 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi064 running (4m) 57s ago 4m 44.5M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b 9adc479ddb88 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi123 running (4m) 57s ago 4m 55.5M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b e701a333bd3e 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 smithi161 running (4m) 57s ago 4m 43.1M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b 3fcdaa503a82 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9 smithi064 running (3m) 57s ago 3m 56.1M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b 9bfbeaeec7d4 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 smithi123 running (3m) 57s ago 3m 53.2M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b 121271c48962 2024-08-19T01:32:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 smithi161 running (2m) 57s ago 2m 41.4M 4272M 19.1.0-1382-gde09dd01 ea7a5189f30b fced39acefc6 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34240a2c40 msgr2=0x7f34240a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34240a2c40 0x7f34240a5100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f34381990f0 tx=0x7f342000a2d0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 msgr2=0x7f3438197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438197bd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3428031610 tx=0x7f34280043b0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 shutdown_connections 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f34240a2c40 0x7f34240a5100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3438102400 0x7f343819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3438101a40 0x7f3438198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 --2- 172.21.15.64:0/3748013231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3438100840 0x7f3438197bd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:47.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 >> 172.21.15.64:0/3748013231 conn(0x7f34380fbfd0 msgr2=0x7f3438104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:47.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 shutdown_connections 2024-08-19T01:32:47.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:47.263+0000 7f343cc52640 1 -- 172.21.15.64:0/3748013231 wait complete. 2024-08-19T01:32:47.966 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:47 smithi064 ceph-mon[18515]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:47.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- bash -c 'ceph orch ls' 2024-08-19T01:32:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:47 smithi123 ceph-mon[21829]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:48.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:47 smithi161 ceph-mon[21757]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:48.250 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:48.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/4242927052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94101930 msgr2=0x7ffa94101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:48.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/4242927052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94101930 0x7ffa94101d30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009900 tx=0x7ffa7c02a410 comp rx=0 tx=0).stop 2024-08-19T01:32:48.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/4242927052 shutdown_connections 2024-08-19T01:32:48.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/4242927052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa941034f0 0x7ffa94109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/4242927052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa94102b30 0x7ffa94102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/4242927052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94101930 0x7ffa94101d30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/4242927052 >> 172.21.15.64:0/4242927052 conn(0x7ffa940fd0c0 msgr2=0x7ffa940ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:48.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/4242927052 shutdown_connections 2024-08-19T01:32:48.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/4242927052 wait complete. 2024-08-19T01:32:48.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.700+0000 7ffa9ad6e640 1 Processor -- start 2024-08-19T01:32:48.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9ad6e640 1 -- start start 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9ad6e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa94101930 0x7ffa94197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9ad6e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94102b30 0x7ffa941982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9ad6e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa941034f0 0x7ffa9419f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9ad6e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9410d690 con 0x7ffa94102b30 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9ad6e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9410d510 con 0x7ffa94101930 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9ad6e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9410d810 con 0x7ffa941034f0 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94102b30 0x7ffa941982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94102b30 0x7ffa941982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35594/0 (socket says 172.21.15.64:35594) 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 -- 172.21.15.64:0/186395891 learned_addr learned my addr 172.21.15.64:0/186395891 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:48.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa98b0a640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa94101930 0x7ffa94197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa9930b640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa941034f0 0x7ffa9419f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 -- 172.21.15.64:0/186395891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa941034f0 msgr2=0x7ffa9419f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa941034f0 0x7ffa9419f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 -- 172.21.15.64:0/186395891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa94101930 msgr2=0x7ffa94197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa94101930 0x7ffa94197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 -- 172.21.15.64:0/186395891 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9419f9c0 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.701+0000 7ffa93fff640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94102b30 0x7ffa941982e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400ba50 tx=0x7ffa8400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.702+0000 7ffa91ffb640 1 -- 172.21.15.64:0/186395891 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84019070 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.702+0000 7ffa91ffb640 1 -- 172.21.15.64:0/186395891 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa8400cbe0 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.702+0000 7ffa91ffb640 1 -- 172.21.15.64:0/186395891 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa840076e0 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.702+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9419fcb0 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.702+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa941a01c0 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.703+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa94108dd0 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.704+0000 7ffa91ffb640 1 -- 172.21.15.64:0/186395891 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ffa8401c070 con 0x7ffa94102b30 2024-08-19T01:32:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.704+0000 7ffa91ffb640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffa800a2c90 0x7ffa800a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:48.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.705+0000 7ffa98b0a640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffa800a2c90 0x7ffa800a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:48.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.705+0000 7ffa91ffb640 1 -- 172.21.15.64:0/186395891 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffa840ce9e0 con 0x7ffa94102b30 2024-08-19T01:32:48.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.705+0000 7ffa98b0a640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffa800a2c90 0x7ffa800a5150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c005ab0 tx=0x7ffa7c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:48.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.709+0000 7ffa91ffb640 1 -- 172.21.15.64:0/186395891 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ffa8409aed0 con 0x7ffa94102b30 2024-08-19T01:32:48.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.893+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ffa94062360 con 0x7ffa800a2c90 2024-08-19T01:32:48.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:48 smithi064 ceph-mon[18515]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:48.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.899+0000 7ffa91ffb640 1 -- 172.21.15.64:0/186395891 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ffa94062360 con 0x7ffa800a2c90 2024-08-19T01:32:48.902 INFO:teuthology.orchestra.run.smithi064.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-08-19T01:32:48.902 INFO:teuthology.orchestra.run.smithi064.stdout:mds.all 12/12 59s ago 2m smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-08-19T01:32:48.902 INFO:teuthology.orchestra.run.smithi064.stdout:mgr 3/3 59s ago 7m smithi064=x;smithi123=y;smithi161=z;count:3 2024-08-19T01:32:48.902 INFO:teuthology.orchestra.run.smithi064.stdout:mon 3/3 59s ago 7m smithi064:172.21.15.64=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c;count:3 2024-08-19T01:32:48.902 INFO:teuthology.orchestra.run.smithi064.stdout:osd 12 59s ago - 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffa800a2c90 msgr2=0x7ffa800a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffa800a2c90 0x7ffa800a5150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c005ab0 tx=0x7ffa7c000bc0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94102b30 msgr2=0x7ffa941982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94102b30 0x7ffa941982e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400ba50 tx=0x7ffa8400bf10 comp rx=0 tx=0).stop 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 shutdown_connections 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7ffa800a2c90 0x7ffa800a5150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa941034f0 0x7ffa9419f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffa94102b30 0x7ffa941982e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 --2- 172.21.15.64:0/186395891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa94101930 0x7ffa94197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:48.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 >> 172.21.15.64:0/186395891 conn(0x7ffa940fd0c0 msgr2=0x7ffa94105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:48.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 shutdown_connections 2024-08-19T01:32:48.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:48.901+0000 7ffa9ad6e640 1 -- 172.21.15.64:0/186395891 wait complete. 2024-08-19T01:32:49.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:48 smithi123 ceph-mon[21829]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:49.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:48 smithi161 ceph-mon[21757]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:49.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- bash -c 'ceph orch host ls' 2024-08-19T01:32:49.938 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:49.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:49 smithi064 ceph-mon[18515]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:49 smithi123 ceph-mon[21829]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:50.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:49 smithi161 ceph-mon[21757]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 -- 172.21.15.64:0/1387951622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c1023d0 msgr2=0x7f1e9c1047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/1387951622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c1023d0 0x7f1e9c1047c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009900 tx=0x7f1e8c02a410 comp rx=0 tx=0).stop 2024-08-19T01:32:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 -- 172.21.15.64:0/1387951622 shutdown_connections 2024-08-19T01:32:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/1387951622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e9c107630 0x7f1e9c109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/1387951622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e9c104d00 0x7f1e9c1070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/1387951622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c1023d0 0x7f1e9c1047c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 -- 172.21.15.64:0/1387951622 >> 172.21.15.64:0/1387951622 conn(0x7f1e9c0fbfd0 msgr2=0x7f1e9c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 -- 172.21.15.64:0/1387951622 shutdown_connections 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 -- 172.21.15.64:0/1387951622 wait complete. 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 Processor -- start 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.419+0000 7f1ea2a05640 1 -- start start 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea2a05640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e9c1023d0 0x7f1e9c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea2a05640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e9c104d00 0x7f1e9c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea2a05640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c107630 0x7f1e9c19f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea2a05640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c10d5e0 con 0x7f1e9c107630 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea2a05640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c10d460 con 0x7f1e9c104d00 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea2a05640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c10d760 con 0x7f1e9c1023d0 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c107630 0x7f1e9c19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c107630 0x7f1e9c19f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35068/0 (socket says 172.21.15.64:35068) 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 -- 172.21.15.64:0/4292383958 learned_addr learned my addr 172.21.15.64:0/4292383958 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:50.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1e93fff640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e9c104d00 0x7f1e9c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1e9bfff640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e9c1023d0 0x7f1e9c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 -- 172.21.15.64:0/4292383958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e9c1023d0 msgr2=0x7f1e9c197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e9c1023d0 0x7f1e9c197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 -- 172.21.15.64:0/4292383958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e9c104d00 msgr2=0x7f1e9c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e9c104d00 0x7f1e9c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 -- 172.21.15.64:0/4292383958 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c19f7f0 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea0fa2640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c107630 0x7f1e9c19f180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400cce0 tx=0x7f1e9400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1e99ffb640 1 -- 172.21.15.64:0/4292383958 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9400f450 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1e99ffb640 1 -- 172.21.15.64:0/4292383958 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e9400fad0 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9c19fae0 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.420+0000 7f1e99ffb640 1 -- 172.21.15.64:0/4292383958 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94018c10 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.421+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9c1a0020 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.422+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9c0fdac0 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.423+0000 7f1e99ffb640 1 -- 172.21.15.64:0/4292383958 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1e9400f5f0 con 0x7f1e9c107630 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.423+0000 7f1e99ffb640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1e880a2cc0 0x7f1e880a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:50.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.423+0000 7f1e99ffb640 1 -- 172.21.15.64:0/4292383958 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1e940cfb80 con 0x7f1e9c107630 2024-08-19T01:32:50.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.423+0000 7f1e9bfff640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1e880a2cc0 0x7f1e880a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:50.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.424+0000 7f1e9bfff640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1e880a2cc0 0x7f1e880a5180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c000c00 tx=0x7f1e8c0315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:50.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.428+0000 7f1e99ffb640 1 -- 172.21.15.64:0/4292383958 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1e94014030 con 0x7f1e9c107630 2024-08-19T01:32:50.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.611+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f1e9c062360 con 0x7f1e880a2cc0 2024-08-19T01:32:50.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.612+0000 7f1e99ffb640 1 -- 172.21.15.64:0/4292383958 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f1e9c062360 con 0x7f1e880a2cc0 2024-08-19T01:32:50.614 INFO:teuthology.orchestra.run.smithi064.stdout:HOST ADDR LABELS STATUS 2024-08-19T01:32:50.614 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 172.21.15.64 2024-08-19T01:32:50.614 INFO:teuthology.orchestra.run.smithi064.stdout:smithi123 172.21.15.123 2024-08-19T01:32:50.614 INFO:teuthology.orchestra.run.smithi064.stdout:smithi161 172.21.15.161 2024-08-19T01:32:50.614 INFO:teuthology.orchestra.run.smithi064.stdout:3 hosts in cluster 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1e880a2cc0 msgr2=0x7f1e880a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1e880a2cc0 0x7f1e880a5180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c000c00 tx=0x7f1e8c0315d0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c107630 msgr2=0x7f1e9c19f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c107630 0x7f1e9c19f180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400cce0 tx=0x7f1e9400a620 comp rx=0 tx=0).stop 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 shutdown_connections 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f1e880a2cc0 0x7f1e880a5180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9c107630 0x7f1e9c19f180 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e9c104d00 0x7f1e9c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 --2- 172.21.15.64:0/4292383958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e9c1023d0 0x7f1e9c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:50.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.614+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 >> 172.21.15.64:0/4292383958 conn(0x7f1e9c0fbfd0 msgr2=0x7f1e9c102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:50.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.615+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 shutdown_connections 2024-08-19T01:32:50.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:50.615+0000 7f1ea2a05640 1 -- 172.21.15.64:0/4292383958 wait complete. 2024-08-19T01:32:50.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:50 smithi064 ceph-mon[18515]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:50.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:50 smithi064 ceph-mon[18515]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:51.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:50 smithi123 ceph-mon[21829]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:51.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:50 smithi123 ceph-mon[21829]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:51.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:50 smithi161 ceph-mon[21757]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:51.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:50 smithi161 ceph-mon[21757]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:51.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- bash -c 'ceph orch device ls' 2024-08-19T01:32:51.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:51 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:32:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:51 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:32:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:51 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:32:52.259 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:52.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 -- 172.21.15.64:0/996033904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 msgr2=0x7f09b8109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:52.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 --2- 172.21.15.64:0/996033904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8109d80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009ca0 tx=0x7f09b002a8c0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 -- 172.21.15.64:0/996033904 shutdown_connections 2024-08-19T01:32:52.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 --2- 172.21.15.64:0/996033904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8109d80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 --2- 172.21.15.64:0/996033904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b8102b30 0x7f09b8102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 --2- 172.21.15.64:0/996033904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b8101930 0x7f09b8101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 -- 172.21.15.64:0/996033904 >> 172.21.15.64:0/996033904 conn(0x7f09b80fd0c0 msgr2=0x7f09b80ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 -- 172.21.15.64:0/996033904 shutdown_connections 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 -- 172.21.15.64:0/996033904 wait complete. 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 Processor -- start 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 -- start start 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.751+0000 7f09bef45640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b8101930 0x7f09b8075cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bef45640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b8102b30 0x7f09b8074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bef45640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bef45640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f09b810d590 con 0x7f09b81034f0 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bef45640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09b810d410 con 0x7f09b8101930 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bef45640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09b810d710 con 0x7f09b8102b30 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8074850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35082/0 (socket says 172.21.15.64:35082) 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09b7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b8102b30 0x7f09b8074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 -- 172.21.15.64:0/3039687658 learned_addr learned my addr 172.21.15.64:0/3039687658 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bcce1640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b8101930 0x7f09b8075cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 -- 172.21.15.64:0/3039687658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b8102b30 msgr2=0x7f09b8074310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b8102b30 0x7f09b8074310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 -- 172.21.15.64:0/3039687658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b8101930 msgr2=0x7f09b8075cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b8101930 0x7f09b8075cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 -- 172.21.15.64:0/3039687658 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b8075110 con 0x7f09b81034f0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bd4e2640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8074850 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f09b0031ae0 tx=0x7f09b0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09b5ffb640 1 -- 172.21.15.64:0/3039687658 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0003c30 con 0x7f09b81034f0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09b5ffb640 1 -- 172.21.15.64:0/3039687658 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b0003dd0 con 0x7f09b81034f0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b81a1cc0 con 0x7f09b81034f0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.752+0000 7f09b5ffb640 1 -- 172.21.15.64:0/3039687658 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b002fcb0 con 0x7f09b81034f0 2024-08-19T01:32:52.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.753+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b81a2180 con 0x7f09b81034f0 2024-08-19T01:32:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.754+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09b8108cb0 con 0x7f09b81034f0 2024-08-19T01:32:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.755+0000 7f09b5ffb640 1 -- 172.21.15.64:0/3039687658 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f09b0004050 con 0x7f09b81034f0 2024-08-19T01:32:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.755+0000 7f09b5ffb640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f09a40a2c10 0x7f09a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.755+0000 7f09b5ffb640 1 -- 172.21.15.64:0/3039687658 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f09b00f06b0 con 0x7f09b81034f0 2024-08-19T01:32:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.755+0000 7f09bcce1640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f09a40a2c10 0x7f09a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.756+0000 7f09bcce1640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f09a40a2c10 0x7f09a40a50d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09a8005fd0 tx=0x7f09a800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.759+0000 7f09b5ffb640 1 -- 172.21.15.64:0/3039687658 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f09b00bd1f0 con 0x7f09b81034f0 2024-08-19T01:32:52.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.948+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f09b8062360 con 0x7f09a40a2c10 2024-08-19T01:32:52.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.955+0000 7f09b5ffb640 1 -- 172.21.15.64:0/3039687658 <== mgr.14150 v2:172.21.15.64:6800/2663741075 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f09b8062360 con 0x7f09a40a2c10 2024-08-19T01:32:52.957 INFO:teuthology.orchestra.run.smithi064.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-08-19T01:32:52.957 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-08-19T01:32:52.957 INFO:teuthology.orchestra.run.smithi064.stdout:smithi123 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400YK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-08-19T01:32:52.957 INFO:teuthology.orchestra.run.smithi064.stdout:smithi161 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233001X400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.957+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f09a40a2c10 msgr2=0x7f09a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.957+0000 7f09bef45640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f09a40a2c10 0x7f09a40a50d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09a8005fd0 tx=0x7f09a800b410 comp rx=0 tx=0).stop 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.957+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 msgr2=0x7f09b8074850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.957+0000 7f09bef45640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8074850 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f09b0031ae0 tx=0x7f09b0031ee0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 shutdown_connections 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f09a40a2c10 0x7f09a40a50d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b81034f0 0x7f09b8074850 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b8102b30 0x7f09b8074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 --2- 172.21.15.64:0/3039687658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b8101930 0x7f09b8075cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:52.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 >> 172.21.15.64:0/3039687658 conn(0x7f09b80fd0c0 msgr2=0x7f09b8105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:52.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 shutdown_connections 2024-08-19T01:32:52.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:52.958+0000 7f09bef45640 1 -- 172.21.15.64:0/3039687658 wait complete. 2024-08-19T01:32:53.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:53 smithi064 ceph-mon[18515]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:53.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:53 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.780 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-08-19T01:32:53.789 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:32:53.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- bash -c 'ceph fs dump' 2024-08-19T01:32:53.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:53 smithi123 ceph-mon[21829]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:53 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:53 smithi161 ceph-mon[21757]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-08-19T01:32:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:53.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:53 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 -- 172.21.15.64:0/3467688005 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378104cc0 msgr2=0x7f73781070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 --2- 172.21.15.64:0/3467688005 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378104cc0 0x7f73781070b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7368009900 tx=0x7f736802a3f0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 -- 172.21.15.64:0/3467688005 shutdown_connections 2024-08-19T01:32:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 --2- 172.21.15.64:0/3467688005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73781075f0 0x7f7378109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 --2- 172.21.15.64:0/3467688005 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378104cc0 0x7f73781070b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 --2- 172.21.15.64:0/3467688005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7378102390 0x7f7378104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 -- 172.21.15.64:0/3467688005 >> 172.21.15.64:0/3467688005 conn(0x7f73780fbfb0 msgr2=0x7f73780fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 -- 172.21.15.64:0/3467688005 shutdown_connections 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.585+0000 7f737f5b4640 1 -- 172.21.15.64:0/3467688005 wait complete. 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 Processor -- start 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 -- start start 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378102390 0x7f7378197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7378104cc0 0x7f7378198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73781075f0 0x7f737819f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f737810d590 con 0x7f7378102390 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f737810d410 con 0x7f73781075f0 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737f5b4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f737810d710 con 0x7f7378104cc0 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378102390 0x7f7378197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378102390 0x7f7378197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35102/0 (socket says 172.21.15.64:35102) 2024-08-19T01:32:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 -- 172.21.15.64:0/2517763794 learned_addr learned my addr 172.21.15.64:0/2517763794 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737db51640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73781075f0 0x7f737819f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737cb4f640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7378104cc0 0x7f7378198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 -- 172.21.15.64:0/2517763794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7378104cc0 msgr2=0x7f7378198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7378104cc0 0x7f7378198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 -- 172.21.15.64:0/2517763794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73781075f0 msgr2=0x7f737819f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73781075f0 0x7f737819f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737d350640 1 -- 172.21.15.64:0/2517763794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737819f850 con 0x7f7378102390 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.586+0000 7f737cb4f640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7378104cc0 0x7f7378198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.587+0000 7f737d350640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378102390 0x7f7378197c30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f737400c920 tx=0x7f737400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.587+0000 7f736e7fc640 1 -- 172.21.15.64:0/2517763794 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374007c70 con 0x7f7378102390 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.587+0000 7f736e7fc640 1 -- 172.21.15.64:0/2517763794 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7374007e10 con 0x7f7378102390 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.587+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f737819fb40 con 0x7f7378102390 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.587+0000 7f736e7fc640 1 -- 172.21.15.64:0/2517763794 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374018890 con 0x7f7378102390 2024-08-19T01:32:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.587+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73781a0080 con 0x7f7378102390 2024-08-19T01:32:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.588+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73780fdaa0 con 0x7f7378102390 2024-08-19T01:32:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.589+0000 7f736e7fc640 1 -- 172.21.15.64:0/2517763794 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7374018a70 con 0x7f7378102390 2024-08-19T01:32:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.589+0000 7f736e7fc640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f73640a2c10 0x7f73640a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.590+0000 7f736e7fc640 1 -- 172.21.15.64:0/2517763794 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f73740cf6f0 con 0x7f7378102390 2024-08-19T01:32:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.590+0000 7f737cb4f640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f73640a2c10 0x7f73640a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.590+0000 7f737cb4f640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f73640a2c10 0x7f73640a50d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7368000c00 tx=0x7f7368031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.594+0000 7f736e7fc640 1 -- 172.21.15.64:0/2517763794 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f7374014030 con 0x7f7378102390 2024-08-19T01:32:54.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.808+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f737802d850 con 0x7f7378102390 2024-08-19T01:32:54.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:54 smithi064 ceph-mon[18515]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:54.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:32:54.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:32:54.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:54 smithi064 ceph-mon[18515]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:54.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.808+0000 7f736e7fc640 1 -- 172.21.15.64:0/2517763794 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3094 (secure 0 0 0) 0x7f737409c120 con 0x7f7378102390 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:e24 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:btime 2024-08-19T01:31:35:584965+0000 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:legacy client fscid: 1 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:Filesystem 'cephfs' (1) 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:fs_name cephfs 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 24 2024-08-19T01:32:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-08-19T01:30:52.282253+0000 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-08-19T01:31:35.353608+0000 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:tableserver 0 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:root 0 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:session_timeout 300 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:session_autoclose 300 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:max_file_size 1099511627776 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:max_xattr_size 65536 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:required_client_features {} 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:last_failure 0 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:last_failure_osd_epoch 0 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:max_mds 3 2024-08-19T01:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:in 0,1,2 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:up {0=24476,1=24452,2=24436} 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:failed 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:damaged 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:stopped 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:data_pools [3,4] 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:metadata_pool 2 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:inline_data disabled 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:balancer 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:bal_rank_mask -1 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:standby_count_wanted 0 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:qdb_cluster leader: 24476 members: 24476,24452,24436 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.b{0:24476} state up:active seq 4 addr [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.e{0:24428} state up:standby-replay seq 1 addr [v2:172.21.15.123:6834/2709386676,v1:172.21.15.123:6835/2709386676] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.h{1:24452} state up:active seq 17 addr [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.c{1:24421} state up:standby-replay seq 1 addr [v2:172.21.15.161:6836/1326350329,v1:172.21.15.161:6837/1326350329] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.813 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.i{2:24436} state up:active seq 17 addr [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.f{2:24415} state up:standby-replay seq 1 addr [v2:172.21.15.161:6834/3530643584,v1:172.21.15.161:6835/3530643584] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:Standby daemons: 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.j{-1:14550} state up:standby seq 1 addr [v2:172.21.15.64:6834/1183738942,v1:172.21.15.64:6835/1183738942] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.d{-1:14562} state up:standby seq 1 addr [v2:172.21.15.64:6836/404621970,v1:172.21.15.64:6837/404621970] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.g{-1:14616} state up:standby seq 1 addr [v2:172.21.15.64:6838/2883191544,v1:172.21.15.64:6839/2883191544] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.a{-1:14622} state up:standby seq 1 addr [v2:172.21.15.64:6840/799479773,v1:172.21.15.64:6841/799479773] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.l{-1:24379} state up:standby seq 1 addr [v2:172.21.15.161:6832/4151238815,v1:172.21.15.161:6833/4151238815] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.814 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.k{-1:24404} state up:standby seq 1 addr [v2:172.21.15.123:6832/856914093,v1:172.21.15.123:6833/856914093] compat {c=[1],r=[1],i=[1fff]}] 2024-08-19T01:32:54.815 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 24 2024-08-19T01:32:54.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f73640a2c10 msgr2=0x7f73640a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:54.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f73640a2c10 0x7f73640a50d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7368000c00 tx=0x7f7368031400 comp rx=0 tx=0).stop 2024-08-19T01:32:54.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378102390 msgr2=0x7f7378197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:54.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378102390 0x7f7378197c30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f737400c920 tx=0x7f737400cde0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 shutdown_connections 2024-08-19T01:32:54.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f73640a2c10 0x7f73640a50d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73781075f0 0x7f737819f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7378104cc0 0x7f7378198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 --2- 172.21.15.64:0/2517763794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7378102390 0x7f7378197c30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:54.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.810+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 >> 172.21.15.64:0/2517763794 conn(0x7f73780fbfb0 msgr2=0x7f7378102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:54.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.811+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 shutdown_connections 2024-08-19T01:32:54.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:54.811+0000 7f737f5b4640 1 -- 172.21.15.64:0/2517763794 wait complete. 2024-08-19T01:32:54.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:54 smithi123 ceph-mon[21829]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:54.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:32:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:32:54.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:54 smithi123 ceph-mon[21829]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:54 smithi161 ceph-mon[21757]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:32:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:32:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:32:54.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:54 smithi161 ceph-mon[21757]: from='mgr.14150 172.21.15.64:0/3601802101' entity='mgr.x' 2024-08-19T01:32:55.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- bash -c 'ceph osd dump' 2024-08-19T01:32:55.820 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:55.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:55 smithi123 ceph-mon[21829]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:55.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:55 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2517763794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-08-19T01:32:55.941 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:55 smithi064 ceph-mon[18515]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:55.942 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:55 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2517763794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-08-19T01:32:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:55 smithi161 ceph-mon[21757]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:32:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:55 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2517763794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-08-19T01:32:56.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 -- 172.21.15.64:0/2306112351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 msgr2=0x7f044c100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 --2- 172.21.15.64:0/2306112351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c100c40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0440009900 tx=0x7f044002a410 comp rx=0 tx=0).stop 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 -- 172.21.15.64:0/2306112351 shutdown_connections 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 --2- 172.21.15.64:0/2306112351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c102400 0x7f044c108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 --2- 172.21.15.64:0/2306112351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f044c101a40 0x7f044c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 --2- 172.21.15.64:0/2306112351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c100c40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 -- 172.21.15.64:0/2306112351 >> 172.21.15.64:0/2306112351 conn(0x7f044c0fbfd0 msgr2=0x7f044c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 -- 172.21.15.64:0/2306112351 shutdown_connections 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.319+0000 7f0453f8b640 1 -- 172.21.15.64:0/2306112351 wait complete. 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 Processor -- start 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 -- start start 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f044c101a40 0x7f044c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c102400 0x7f044c19f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:56.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f044c10d6b0 con 0x7f044c100840 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f044c10d530 con 0x7f044c102400 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0453f8b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f044c10d830 con 0x7f044c101a40 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35126/0 (socket says 172.21.15.64:35126) 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 -- 172.21.15.64:0/895127130 learned_addr learned my addr 172.21.15.64:0/895127130 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0452528640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c102400 0x7f044c19f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 -- 172.21.15.64:0/895127130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f044c101a40 msgr2=0x7f044c198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f044c101a40 0x7f044c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 -- 172.21.15.64:0/895127130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c102400 msgr2=0x7f044c19f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c102400 0x7f044c19f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 -- 172.21.15.64:0/895127130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044c19f9f0 con 0x7f044c100840 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0452528640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c102400 0x7f044c19f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:56.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.320+0000 7f0451d27640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c197dc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0440031610 tx=0x7f04400043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.321+0000 7f043effd640 1 -- 172.21.15.64:0/895127130 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044003f030 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.321+0000 7f043effd640 1 -- 172.21.15.64:0/895127130 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0440003ce0 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.321+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044c19fc80 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.321+0000 7f043effd640 1 -- 172.21.15.64:0/895127130 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440035070 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.321+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044c1a0160 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.322+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044c107d30 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.323+0000 7f043effd640 1 -- 172.21.15.64:0/895127130 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0440042030 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.323+0000 7f043effd640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04380a2c40 0x7f04380a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.324+0000 7f0451526640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04380a2c40 0x7f04380a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.324+0000 7f043effd640 1 -- 172.21.15.64:0/895127130 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f04400eff00 con 0x7f044c100840 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.324+0000 7f0451526640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04380a2c40 0x7f04380a5100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f044c1992e0 tx=0x7f0448006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.327+0000 7f043effd640 1 -- 172.21.15.64:0/895127130 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f04400bc930 con 0x7f044c100840 2024-08-19T01:32:56.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.504+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f044c02d780 con 0x7f044c100840 2024-08-19T01:32:56.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.505+0000 7f043effd640 1 -- 172.21.15.64:0/895127130 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5701 (secure 0 0 0) 0x7f04400bc230 con 0x7f044c100840 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 93 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-08-19T01:23:48.740738+0000 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-08-19T01:32:35.645177+0000 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:crush_version 27 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:full_ratio 0.95 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:backfillfull_ratio 0.9 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:nearfull_ratio 0.85 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:require_min_compat_client luminous 2024-08-19T01:32:56.507 INFO:teuthology.orchestra.run.smithi064.stdout:min_compat_client luminous 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:require_osd_release squid 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:stretch_mode_enabled false 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:max_osd 12 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6802/3320376368,v1:172.21.15.64:6803/3320376368] [v2:172.21.15.64:6804/3320376368,v1:172.21.15.64:6805/3320376368] exists,up 8c3fb9c1-436e-47fd-ae54-4f88a7b757c0 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6800/4102727954,v1:172.21.15.123:6801/4102727954] [v2:172.21.15.123:6802/4102727954,v1:172.21.15.123:6803/4102727954] exists,up 7e1f4596-0fe3-4857-a061-1b173ea09ea3 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6800/2169170995,v1:172.21.15.161:6801/2169170995] [v2:172.21.15.161:6802/2169170995,v1:172.21.15.161:6803/2169170995] exists,up b86cd6b3-a20f-4b12-82ca-c926be9cf348 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 up in weight 1 up_from 24 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6810/18842482,v1:172.21.15.64:6811/18842482] [v2:172.21.15.64:6812/18842482,v1:172.21.15.64:6813/18842482] exists,up c9312a92-2ed5-4111-8e98-1822e682fe2c 2024-08-19T01:32:56.508 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 up in weight 1 up_from 31 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6808/529776515,v1:172.21.15.123:6809/529776515] [v2:172.21.15.123:6810/529776515,v1:172.21.15.123:6811/529776515] exists,up eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 up in weight 1 up_from 36 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6808/3335610027,v1:172.21.15.161:6809/3335610027] [v2:172.21.15.161:6810/3335610027,v1:172.21.15.161:6811/3335610027] exists,up 3566d78c-0daf-4391-bd31-1f8c95f93d35 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 up in weight 1 up_from 43 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6818/3144814721,v1:172.21.15.64:6819/3144814721] [v2:172.21.15.64:6820/3144814721,v1:172.21.15.64:6821/3144814721] exists,up 727df6d7-b958-434a-9812-a1b798a99b8f 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 up in weight 1 up_from 48 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6816/2282165351,v1:172.21.15.123:6817/2282165351] [v2:172.21.15.123:6818/2282165351,v1:172.21.15.123:6819/2282165351] exists,up 30066775-0caa-4862-bd5b-45711ad1c571 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 up in weight 1 up_from 56 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6816/2639499139,v1:172.21.15.161:6817/2639499139] [v2:172.21.15.161:6818/2639499139,v1:172.21.15.161:6819/2639499139] exists,up c5b64329-e576-4c16-87ae-2180430f9ca5 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9 up in weight 1 up_from 64 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6826/1073086660,v1:172.21.15.64:6827/1073086660] [v2:172.21.15.64:6828/1073086660,v1:172.21.15.64:6829/1073086660] exists,up 50795cb0-f711-4f52-b8fa-e9583c165cae 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 up in weight 1 up_from 69 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6824/3299707057,v1:172.21.15.123:6825/3299707057] [v2:172.21.15.123:6826/3299707057,v1:172.21.15.123:6827/3299707057] exists,up fe117900-4a44-493a-b389-37c19ce9356a 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 up in weight 1 up_from 77 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6824/564834779,v1:172.21.15.161:6825/564834779] [v2:172.21.15.161:6826/564834779,v1:172.21.15.161:6827/564834779] exists,up 3d23aa4f-abcc-4207-aa16-d6e3975b613f 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.b [8,11] 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.14 [2,4] 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.1b [2,1] 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.1d [8,11] 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.0 [5,9] 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.3 [4,9] 2024-08-19T01:32:56.509 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.6 [3,6] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.17 [4,8] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.1b [4,7] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.37 [1,9] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.7 [5,8] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.c [9,10] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.13 [9,8] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.16 [9,8] 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3202507182 expires 2024-08-20T01:24:33.597653+0000 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/2403632526 expires 2024-08-20T01:24:33.597653+0000 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1948699247 expires 2024-08-20T01:24:10.525240+0000 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1182511186 expires 2024-08-20T01:24:10.525240+0000 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/4090416923 expires 2024-08-20T01:24:33.597653+0000 2024-08-19T01:32:56.510 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/973985936 expires 2024-08-20T01:24:10.525240+0000 2024-08-19T01:32:56.511 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6801/3149805608 expires 2024-08-20T01:24:10.525240+0000 2024-08-19T01:32:56.511 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6801/2403632526 expires 2024-08-20T01:24:33.597653+0000 2024-08-19T01:32:56.511 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3838744191 expires 2024-08-20T01:24:33.597653+0000 2024-08-19T01:32:56.511 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/3149805608 expires 2024-08-20T01:24:10.525240+0000 2024-08-19T01:32:56.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04380a2c40 msgr2=0x7f04380a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:56.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04380a2c40 0x7f04380a5100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f044c1992e0 tx=0x7f0448006c90 comp rx=0 tx=0).stop 2024-08-19T01:32:56.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 msgr2=0x7f044c197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c197dc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0440031610 tx=0x7f04400043b0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 shutdown_connections 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04380a2c40 0x7f04380a5100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c102400 0x7f044c19f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f044c101a40 0x7f044c198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 --2- 172.21.15.64:0/895127130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f044c100840 0x7f044c197dc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 >> 172.21.15.64:0/895127130 conn(0x7f044c0fbfd0 msgr2=0x7f044c104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 shutdown_connections 2024-08-19T01:32:56.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:56.507+0000 7f0453f8b640 1 -- 172.21.15.64:0/895127130 wait complete. 2024-08-19T01:32:56.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:56 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/895127130' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-19T01:32:56.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:56 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/895127130' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-19T01:32:56.967 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:56 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/895127130' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-19T01:32:57.171 INFO:teuthology.run_tasks:Running task fs.ready... 2024-08-19T01:32:57.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:32:57.439 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:57.874 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:57 smithi161 ceph-mon[21757]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:57.885 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:57 smithi064 ceph-mon[18515]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 -- 172.21.15.64:0/147953292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841007b0 msgr2=0x7fc384100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/147953292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841007b0 0x7fc384100c10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc378009930 tx=0x7fc37802a420 comp rx=0 tx=0).stop 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 -- 172.21.15.64:0/147953292 shutdown_connections 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/147953292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc384101150 0x7fc38410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/147953292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841007b0 0x7fc384100c10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/147953292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3841067b0 0x7fc384106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 -- 172.21.15.64:0/147953292 >> 172.21.15.64:0/147953292 conn(0x7fc3840fc4d0 msgr2=0x7fc3840fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 -- 172.21.15.64:0/147953292 shutdown_connections 2024-08-19T01:32:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 -- 172.21.15.64:0/147953292 wait complete. 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.884+0000 7fc38c7ee640 1 Processor -- start 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38c7ee640 1 -- start start 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38c7ee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3841007b0 0x7fc3841980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38c7ee640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc384101150 0x7fc384198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38c7ee640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841067b0 0x7fc38419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38c7ee640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc384110b70 con 0x7fc3841067b0 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38c7ee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841109f0 con 0x7fc3841007b0 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38c7ee640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc384110cf0 con 0x7fc384101150 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841067b0 0x7fc38419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841067b0 0x7fc38419c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35142/0 (socket says 172.21.15.64:35142) 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 -- 172.21.15.64:0/2251865621 learned_addr learned my addr 172.21.15.64:0/2251865621 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc389d89640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc384101150 0x7fc384198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38a58a640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3841007b0 0x7fc3841980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 -- 172.21.15.64:0/2251865621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc384101150 msgr2=0x7fc384198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc384101150 0x7fc384198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 -- 172.21.15.64:0/2251865621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3841007b0 msgr2=0x7fc3841980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3841007b0 0x7fc3841980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.885+0000 7fc38ad8b640 1 -- 172.21.15.64:0/2251865621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38419d0a0 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.886+0000 7fc38ad8b640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841067b0 0x7fc38419c9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00d750 tx=0x7fc37c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.886+0000 7fc3837fe640 1 -- 172.21.15.64:0/2251865621 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c017070 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.886+0000 7fc3837fe640 1 -- 172.21.15.64:0/2251865621 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c005500 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.886+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38419d390 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.886+0000 7fc3837fe640 1 -- 172.21.15.64:0/2251865621 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c004d90 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.886+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3841a4c70 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.887+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc384100c10 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.888+0000 7fc3837fe640 1 -- 172.21.15.64:0/2251865621 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc37c004f30 con 0x7fc3841067b0 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.888+0000 7fc3837fe640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fc36c0a2c10 0x7fc36c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:57.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.889+0000 7fc38a58a640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fc36c0a2c10 0x7fc36c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:57.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.889+0000 7fc3837fe640 1 -- 172.21.15.64:0/2251865621 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc37c0cef30 con 0x7fc3841067b0 2024-08-19T01:32:57.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.889+0000 7fc38a58a640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fc36c0a2c10 0x7fc36c0a50d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc374007e40 tx=0x7fc3740075b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:57.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:57.893+0000 7fc3837fe640 1 -- 172.21.15.64:0/2251865621 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fc37c09b310 con 0x7fc3841067b0 2024-08-19T01:32:57.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:57 smithi123 ceph-mon[21829]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:32:58.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.104+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc384109d50 con 0x7fc3841067b0 2024-08-19T01:32:58.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.105+0000 7fc3837fe640 1 -- 172.21.15.64:0/2251865621 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11921 (secure 0 0 0) 0x7fc37c09f610 con 0x7fc3841067b0 2024-08-19T01:32:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:58.106 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"btime":"2024-08-19T01:31:35:584965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:35.353608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:32:58.107 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 24 2024-08-19T01:32:58.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fc36c0a2c10 msgr2=0x7fc36c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:58.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fc36c0a2c10 0x7fc36c0a50d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc374007e40 tx=0x7fc3740075b0 comp rx=0 tx=0).stop 2024-08-19T01:32:58.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841067b0 msgr2=0x7fc38419c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:58.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841067b0 0x7fc38419c9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00d750 tx=0x7fc37c00dc10 comp rx=0 tx=0).stop 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 shutdown_connections 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fc36c0a2c10 0x7fc36c0a50d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3841067b0 0x7fc38419c9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc384101150 0x7fc384198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 --2- 172.21.15.64:0/2251865621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3841007b0 0x7fc3841980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 >> 172.21.15.64:0/2251865621 conn(0x7fc3840fc4d0 msgr2=0x7fc38406a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 shutdown_connections 2024-08-19T01:32:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:58.107+0000 7fc38c7ee640 1 -- 172.21.15.64:0/2251865621 wait complete. 2024-08-19T01:32:58.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:58 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2251865621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:58.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:58 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2251865621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:58.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:58 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2251865621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:32:59.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:32:59.294 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:32:59.690 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:32:59 smithi064 ceph-mon[18515]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 -- 172.21.15.64:0/60846347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 msgr2=0x7f919c100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 --2- 172.21.15.64:0/60846347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c100c10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9184009930 tx=0x7f918402a420 comp rx=0 tx=0).stop 2024-08-19T01:32:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 -- 172.21.15.64:0/60846347 shutdown_connections 2024-08-19T01:32:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 --2- 172.21.15.64:0/60846347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c101150 0x7f919c10d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 --2- 172.21.15.64:0/60846347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c100c10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 --2- 172.21.15.64:0/60846347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f919c1067b0 0x7f919c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 -- 172.21.15.64:0/60846347 >> 172.21.15.64:0/60846347 conn(0x7f919c0fc4d0 msgr2=0x7f919c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 -- 172.21.15.64:0/60846347 shutdown_connections 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.769+0000 7f91a1828640 1 -- 172.21.15.64:0/60846347 wait complete. 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 Processor -- start 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 -- start start 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c0fca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c101150 0x7f919c0fcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f919c1067b0 0x7f919c0fd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f919c110bd0 con 0x7f919c1007b0 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f919c110a50 con 0x7f919c101150 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f91a1828640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f919c110d50 con 0x7f919c1067b0 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f919affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c0fca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f919affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c0fca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44084/0 (socket says 172.21.15.64:44084) 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.770+0000 7f919affd640 1 -- 172.21.15.64:0/2304835302 learned_addr learned my addr 172.21.15.64:0/2304835302 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:32:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919a7fc640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c101150 0x7f919c0fcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919affd640 1 -- 172.21.15.64:0/2304835302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f919c1067b0 msgr2=0x7f919c0fd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919affd640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f919c1067b0 0x7f919c0fd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919affd640 1 -- 172.21.15.64:0/2304835302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c101150 msgr2=0x7f919c0fcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919affd640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c101150 0x7f919c0fcfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919affd640 1 -- 172.21.15.64:0/2304835302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c1a66a0 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919a7fc640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c101150 0x7f919c0fcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f919affd640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c0fca80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f919400c920 tx=0x7f919400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f91a0826640 1 -- 172.21.15.64:0/2304835302 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194007c70 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f91a0826640 1 -- 172.21.15.64:0/2304835302 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9194007e10 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919c1a6990 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f91a0826640 1 -- 172.21.15.64:0/2304835302 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194018890 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.771+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919c1a6f20 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.772+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f919c100c10 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.773+0000 7f91a0826640 1 -- 172.21.15.64:0/2304835302 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9194018a70 con 0x7f919c1007b0 2024-08-19T01:32:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.774+0000 7f91a0826640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f91880a2c90 0x7f91880a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:32:59.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.774+0000 7f919a7fc640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f91880a2c90 0x7f91880a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:32:59.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.774+0000 7f91a0826640 1 -- 172.21.15.64:0/2304835302 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f91940cf7d0 con 0x7f919c1007b0 2024-08-19T01:32:59.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.775+0000 7f919a7fc640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f91880a2c90 0x7f91880a5150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9184000c00 tx=0x7f9184032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:32:59.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.778+0000 7f91a0826640 1 -- 172.21.15.64:0/2304835302 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f9194014030 con 0x7f919c1007b0 2024-08-19T01:32:59.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:32:59 smithi123 ceph-mon[21829]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:32:59 smithi161 ceph-mon[21757]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:32:59.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.991+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f919c109d80 con 0x7f919c1007b0 2024-08-19T01:32:59.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.992+0000 7f91a0826640 1 -- 172.21.15.64:0/2304835302 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11921 (secure 0 0 0) 0x7f919409c1a0 con 0x7f919c1007b0 2024-08-19T01:32:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:32:59.993 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"btime":"2024-08-19T01:31:35:584965+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:31:35.353608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:32:59.994 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 24 2024-08-19T01:32:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f91880a2c90 msgr2=0x7f91880a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f91880a2c90 0x7f91880a5150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9184000c00 tx=0x7f9184032d50 comp rx=0 tx=0).stop 2024-08-19T01:32:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 msgr2=0x7f919c0fca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:32:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c0fca80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f919400c920 tx=0x7f919400cde0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 shutdown_connections 2024-08-19T01:32:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f91880a2c90 0x7f91880a5150 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f919c1067b0 0x7f919c0fd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c101150 0x7f919c0fcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 --2- 172.21.15.64:0/2304835302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f919c1007b0 0x7f919c0fca80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:32:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 >> 172.21.15.64:0/2304835302 conn(0x7f919c0fc4d0 msgr2=0x7f919c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:32:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 shutdown_connections 2024-08-19T01:32:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:32:59.994+0000 7f91a1828640 1 -- 172.21.15.64:0/2304835302 wait complete. 2024-08-19T01:33:00.871 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:33:00.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:00 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2304835302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:00.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:00 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2304835302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:00.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:00 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2304835302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:01.129 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:01.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.580+0000 7fe5a93de640 1 -- 172.21.15.64:0/3488415619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41067b0 msgr2=0x7fe5a4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:01.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.580+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3488415619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41067b0 0x7fe5a4106b90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe594009930 tx=0x7fe59402a420 comp rx=0 tx=0).stop 2024-08-19T01:33:01.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.580+0000 7fe5a93de640 1 -- 172.21.15.64:0/3488415619 shutdown_connections 2024-08-19T01:33:01.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.580+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3488415619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5a4101150 0x7fe5a410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.580+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3488415619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a41007b0 0x7fe5a4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.580+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3488415619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41067b0 0x7fe5a4106b90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.580+0000 7fe5a93de640 1 -- 172.21.15.64:0/3488415619 >> 172.21.15.64:0/3488415619 conn(0x7fe5a40fc4d0 msgr2=0x7fe5a40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.581+0000 7fe5a93de640 1 -- 172.21.15.64:0/3488415619 shutdown_connections 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.581+0000 7fe5a93de640 1 -- 172.21.15.64:0/3488415619 wait complete. 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.581+0000 7fe5a93de640 1 Processor -- start 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.581+0000 7fe5a93de640 1 -- start start 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a93de640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41007b0 0x7fe5a419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a93de640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a4101150 0x7fe5a419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a93de640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5a41067b0 0x7fe5a419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a93de640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a4110c00 con 0x7fe5a41007b0 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a93de640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a4110a80 con 0x7fe5a4101150 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a93de640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a4110d80 con 0x7fe5a41067b0 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41007b0 0x7fe5a419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:01.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41007b0 0x7fe5a419a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44112/0 (socket says 172.21.15.64:44112) 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 -- 172.21.15.64:0/3698525164 learned_addr learned my addr 172.21.15.64:0/3698525164 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe59a7fc640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a4101150 0x7fe5a419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a37fe640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5a41067b0 0x7fe5a419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 -- 172.21.15.64:0/3698525164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5a41067b0 msgr2=0x7fe5a419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5a41067b0 0x7fe5a419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 -- 172.21.15.64:0/3698525164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a4101150 msgr2=0x7fe5a419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a4101150 0x7fe5a419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 -- 172.21.15.64:0/3698525164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a419f320 con 0x7fe5a41007b0 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a37fe640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5a41067b0 0x7fe5a419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.582+0000 7fe5a2ffd640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41007b0 0x7fe5a419a370 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe59402a940 tx=0x7fe594003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.583+0000 7fe5a0ff9640 1 -- 172.21.15.64:0/3698525164 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59403f070 con 0x7fe5a41007b0 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.583+0000 7fe5a0ff9640 1 -- 172.21.15.64:0/3698525164 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe594031c90 con 0x7fe5a41007b0 2024-08-19T01:33:01.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.583+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a419f5b0 con 0x7fe5a41007b0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.583+0000 7fe5a0ff9640 1 -- 172.21.15.64:0/3698525164 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe594035030 con 0x7fe5a41007b0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.583+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5a41a6ef0 con 0x7fe5a41007b0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.584+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5a4101ef0 con 0x7fe5a41007b0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.585+0000 7fe5a0ff9640 1 -- 172.21.15.64:0/3698525164 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe594042070 con 0x7fe5a41007b0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.586+0000 7fe5a0ff9640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5900a2c10 0x7fe5900a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.586+0000 7fe59a7fc640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5900a2c10 0x7fe5900a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.586+0000 7fe5a0ff9640 1 -- 172.21.15.64:0/3698525164 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe5940efd40 con 0x7fe5a41007b0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.586+0000 7fe59a7fc640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5900a2c10 0x7fe5900a50d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40fe8a0 tx=0x7fe58c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:01.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.590+0000 7fe5a0ff9640 1 -- 172.21.15.64:0/3698525164 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe5940bc770 con 0x7fe5a41007b0 2024-08-19T01:33:01.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.767+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5a41a70c0 con 0x7fe5a41007b0 2024-08-19T01:33:01.771 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:01 smithi064 ceph-mon[18515]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:01.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.770+0000 7fe5a0ff9640 1 -- 172.21.15.64:0/3698525164 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24830 (secure 0 0 0) 0x7fe5940bc070 con 0x7fe5a41007b0 2024-08-19T01:33:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:01.772 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":93,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:32:35.645177+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:33:01.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5900a2c10 msgr2=0x7fe5900a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5900a2c10 0x7fe5900a50d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40fe8a0 tx=0x7fe58c00a2d0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41007b0 msgr2=0x7fe5a419a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41007b0 0x7fe5a419a370 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe59402a940 tx=0x7fe594003b90 comp rx=0 tx=0).stop 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 shutdown_connections 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe5900a2c10 0x7fe5900a50d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5a41067b0 0x7fe5a419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a4101150 0x7fe5a419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 --2- 172.21.15.64:0/3698525164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5a41007b0 0x7fe5a419a370 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 >> 172.21.15.64:0/3698525164 conn(0x7fe5a40fc4d0 msgr2=0x7fe5a406a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 shutdown_connections 2024-08-19T01:33:01.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:01.772+0000 7fe5a93de640 1 -- 172.21.15.64:0/3698525164 wait complete. 2024-08-19T01:33:01.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:01 smithi123 ceph-mon[21829]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:01.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:01 smithi161 ceph-mon[21757]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:02.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-08-19T01:30:52.282253+0000', 'modified': '2024-08-19T01:31:35.353608+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24476, 'mds_1': 24452, 'mds_2': 24436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24415': {'gid': 24415, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/3530643584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3530643584}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 3530643584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6837/1326350329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 1326350329}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1326350329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6835/2709386676', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 2709386676}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2709386676}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.161:6839/2198702470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2198702470}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2198702470}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.123:6837/3425782983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6836', 'nonce': 3425782983}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 3425782983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/2281975354', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 2281975354}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 2281975354}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24476, 'qdb_cluster': [24476, 24452, 24436]} 2024-08-19T01:33:02.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-08-19T01:33:02.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.i status 2024-08-19T01:33:02.881 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:02 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3698525164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:33:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:02.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:02 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3698525164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:33:02.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:02 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3698525164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:33:03.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 -- 172.21.15.64:0/1383566995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe868100b50 msgr2=0x7fe868100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 --2- 172.21.15.64:0/1383566995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe868100b50 0x7fe868100f30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe858009900 tx=0x7fe85802a480 comp rx=0 tx=0).stop 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 -- 172.21.15.64:0/1383566995 shutdown_connections 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 --2- 172.21.15.64:0/1383566995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe868102c20 0x7fe86810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 --2- 172.21.15.64:0/1383566995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8681022a0 0x7fe8681026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 --2- 172.21.15.64:0/1383566995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe868100b50 0x7fe868100f30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 -- 172.21.15.64:0/1383566995 >> 172.21.15.64:0/1383566995 conn(0x7fe8680fc500 msgr2=0x7fe8680fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 -- 172.21.15.64:0/1383566995 shutdown_connections 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.393+0000 7fe86e761640 1 -- 172.21.15.64:0/1383566995 wait complete. 2024-08-19T01:33:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 Processor -- start 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 -- start start 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe868100b50 0x7fe868198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8681022a0 0x7fe868198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe868102c20 0x7fe86819cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe868110cd0 con 0x7fe8681022a0 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe868110b50 con 0x7fe868100b50 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe86e761640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe868110e50 con 0x7fe868102c20 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8681022a0 0x7fe868198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8681022a0 0x7fe868198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44132/0 (socket says 172.21.15.64:44132) 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.394+0000 7fe85ffff640 1 -- 172.21.15.64:0/3128699898 learned_addr learned my addr 172.21.15.64:0/3128699898 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe868102c20 0x7fe86819cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe867fff640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe868100b50 0x7fe868198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe85ffff640 1 -- 172.21.15.64:0/3128699898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe868102c20 msgr2=0x7fe86819cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe85ffff640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe868102c20 0x7fe86819cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe85ffff640 1 -- 172.21.15.64:0/3128699898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe868100b50 msgr2=0x7fe868198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe85ffff640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe868100b50 0x7fe868198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe85ffff640 1 -- 172.21.15.64:0/3128699898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86819d1b0 con 0x7fe8681022a0 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe867fff640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe868100b50 0x7fe868198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe85ffff640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8681022a0 0x7fe868198740 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe85000c920 tx=0x7fe85000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:03.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe865ffb640 1 -- 172.21.15.64:0/3128699898 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850007c70 con 0x7fe8681022a0 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86819d4a0 con 0x7fe8681022a0 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe865ffb640 1 -- 172.21.15.64:0/3128699898 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe850007e10 con 0x7fe8681022a0 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe865ffb640 1 -- 172.21.15.64:0/3128699898 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850018890 con 0x7fe8681022a0 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.395+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8681a4d80 con 0x7fe8681022a0 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe865ffb640 1 -- 172.21.15.64:0/3128699898 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe850018a70 con 0x7fe8681022a0 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 Processor -- start 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 -- start start 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbd3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbd3f50 0x556becbde2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbde810 0x556becbe0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556becb38710 con 0x556becbd3630 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x556becad0770 con 0x556becbde810 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe870b5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556becb346d0 con 0x556becbd3f50 2024-08-19T01:33:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe86ccfe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbd3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe86ccfe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbd3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44146/0 (socket says 172.21.15.64:44146) 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe86ccfe640 1 -- 172.21.15.64:0/2185853114 learned_addr learned my addr 172.21.15.64:0/2185853114 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe865ffb640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8540a2c10 0x7fe8540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.398+0000 7fe867fff640 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbd3f50 0x556becbde2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8540a2c10 0x7fe8540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe865ffb640 1 -- 172.21.15.64:0/3128699898 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe8500cf6f0 con 0x7fe8681022a0 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe85ffff640 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbde810 0x556becbe0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe86ccfe640 1 -- 172.21.15.64:0/2185853114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbd3f50 msgr2=0x556becbde2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbd3f50 0x556becbde2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe86ccfe640 1 -- 172.21.15.64:0/2185853114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbde810 msgr2=0x556becbe0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbde810 0x556becbe0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe86ccfe640 1 -- 172.21.15.64:0/2185853114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556becad73e0 con 0x556becbd3630 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbd3a10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe86000a990 tx=0x7fe86000ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56933e8c5fd0a38 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:03.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe85e7fc640 1 -- 172.21.15.64:0/2185853114 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86001d030 con 0x556becbd3630 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe85e7fc640 1 -- 172.21.15.64:0/2185853114 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe860015c60 con 0x556becbd3630 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.399+0000 7fe85e7fc640 1 -- 172.21.15.64:0/2185853114 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600213c0 con 0x556becbd3630 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 -- 172.21.15.64:0/2185853114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 msgr2=0x556becbd3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8540a2c10 0x7fe8540a50d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe860020000 tx=0x7fe86001e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbd3a10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe86000a990 tx=0x7fe86000ae50 comp rx=0 tx=0).stop 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 -- 172.21.15.64:0/2185853114 shutdown_connections 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbde810 0x556becbe0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbd3f50 0x556becbde2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2185853114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbd3a10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 -- 172.21.15.64:0/2185853114 >> 172.21.15.64:0/2185853114 conn(0x556becbcfbc0 msgr2=0x556becbcffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 -- 172.21.15.64:0/2185853114 shutdown_connections 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 -- 172.21.15.64:0/2185853114 wait complete. 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.400+0000 7fe870b5f740 1 -- start start 2024-08-19T01:33:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 1 Processor -- start 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbdb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbdbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbdc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556becbe13e0 con 0x556becbd3f50 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x556becaf3970 con 0x556becbd3630 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556becbe1560 con 0x556becbde810 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe870b5f740 10 client.0 _ll_register_callbacks cb 0x556bec8477b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbdbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbdbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44156/0 (socket says 172.21.15.64:44156) 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 -- 172.21.15.64:0/2546377533 learned_addr learned my addr 172.21.15.64:0/2546377533 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbdc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe867fff640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbdb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 -- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 msgr2=0x556becbdc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbdc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 -- 172.21.15.64:0/2546377533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 msgr2=0x556becbdb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbdb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe85ffff640 1 -- 172.21.15.64:0/2546377533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bec8aa2b0 con 0x556becbd3f50 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.401+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbdc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe85ffff640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbdbc40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe8500ccd00 tx=0x7fe85009ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887a225a72e49532 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe8477fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe8477fe640 1 -- 172.21.15.64:0/2546377533 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8500a4690 con 0x556becbd3f50 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe8477fe640 1 -- 172.21.15.64:0/2546377533 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe85000fb30 con 0x556becbd3f50 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe860015c60 con 0x556becbd3f50 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe8477fe640 1 -- 172.21.15.64:0/2546377533 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe8500cf3e0 con 0x556becbd3f50 2024-08-19T01:33:03.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 10 client.15192 fetch_fsmap learned FSMap version 24 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556becbed0a0 con 0x556becbd3f50 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe8477fe640 1 -- 172.21.15.64:0/2546377533 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe860015c60 con 0x556becbd3f50 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe8477fe640 1 -- 172.21.15.64:0/2546377533 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fe8500ba2b0 con 0x556becbd3f50 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 10 client.15192 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 10 client.15192 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 20 client.15192 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becbef680 0x556becbf0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 4 client.15192 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.402+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 --> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556becb45c80 con 0x556becbef680 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.403+0000 7fe867fff640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becbef680 0x556becbf0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.403+0000 7fe8477fe640 10 client.15192 ms_handle_connect on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.403+0000 7fe867fff640 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becbef680 0x556becbf0af0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3e3b548c59a2bf0 server_cookie=6e624e173662111e in_seq=0 out_seq=0 2024-08-19T01:33:03.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe8477fe640 1 -- 172.21.15.64:0/2546377533 <== mds.2 v2:172.21.15.161:6838/2198702470 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556becb45c80 con 0x556becbef680 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe8477fe640 10 client.15192 handle_command_reply: tid=0 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 client.15192 shutdown 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 msgr2=0x556becbdbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbdbc40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe8500ccd00 tx=0x7fe85009ba20 comp rx=0 tx=0).stop 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 shutdown_connections 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becbef680 0x556becbf0af0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbdc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbdbc40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe8477fe640 0 client.15192 ms_handle_reset on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2546377533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbdb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 >> 172.21.15.64:0/2546377533 conn(0x556becbcfbc0 msgr2=0x556becbced40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:03.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 shutdown_connections 2024-08-19T01:33:03.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 1 -- 172.21.15.64:0/2546377533 wait complete. 2024-08-19T01:33:03.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.404+0000 7fe870b5f740 20 client.15192 trim_cache size 0 max 16384 2024-08-19T01:33:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.427+0000 7fe870b5f740 1 Processor -- start 2024-08-19T01:33:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe870b5f740 1 -- start start 2024-08-19T01:33:03.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbdafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3f50 0x556becbda6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe870b5f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556becbeee30 con 0x556becbd3630 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe870b5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x556becbef500 con 0x556becbd3f50 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe870b5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556becbefbd0 con 0x556becbde810 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbdafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbdafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44164/0 (socket says 172.21.15.64:44164) 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 -- 172.21.15.64:0/1592247912 learned_addr learned my addr 172.21.15.64:0/1592247912 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe85ffff640 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3f50 0x556becbda6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 -- 172.21.15.64:0/1592247912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 msgr2=0x556becbd8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 -- 172.21.15.64:0/1592247912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3f50 msgr2=0x556becbda6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3f50 0x556becbda6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe867fff640 1 -- 172.21.15.64:0/1592247912 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556becbcb130 con 0x556becbd3630 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.428+0000 7fe85ffff640 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3f50 0x556becbda6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.429+0000 7fe867fff640 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbdafc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe8580314e0 tx=0x7fe858005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b857f5df4c413b63 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.429+0000 7fe85dffb640 1 -- 172.21.15.64:0/1592247912 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858048030 con 0x556becbd3630 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.429+0000 7fe85dffb640 1 -- 172.21.15.64:0/1592247912 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe858005500 con 0x556becbd3630 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.429+0000 7fe85dffb640 1 -- 172.21.15.64:0/1592247912 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858035040 con 0x556becbd3630 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 -- 172.21.15.64:0/1592247912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 msgr2=0x556becbdafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbdafc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe8580314e0 tx=0x7fe858005b50 comp rx=0 tx=0).stop 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 -- 172.21.15.64:0/1592247912 shutdown_connections 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3f50 0x556becbda6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 --2- 172.21.15.64:0/1592247912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3630 0x556becbdafc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 -- 172.21.15.64:0/1592247912 >> 172.21.15.64:0/1592247912 conn(0x556becbcfbc0 msgr2=0x556becbd9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 -- 172.21.15.64:0/1592247912 shutdown_connections 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 -- 172.21.15.64:0/1592247912 wait complete. 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.430+0000 7fe870b5f740 1 -- start start 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 1 Processor -- start 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbf9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbd8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556becbf0970 con 0x556becbd3f50 2024-08-19T01:33:03.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbd8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x556becbf02a0 con 0x556becbd3630 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556becbf1040 con 0x556becbde810 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe870b5f740 10 client.0 _ll_register_callbacks cb 0x7fe85000cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbd8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44170/0 (socket says 172.21.15.64:44170) 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe85ffff640 1 -- 172.21.15.64:0/2731366546 learned_addr learned my addr 172.21.15.64:0/2731366546 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe867fff640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbf9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe86ccfe640 1 -- 172.21.15.64:0/2731366546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 msgr2=0x556becbf9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.431+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbf9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe86ccfe640 1 -- 172.21.15.64:0/2731366546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 msgr2=0x556becbd8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbd8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe86ccfe640 1 -- 172.21.15.64:0/2731366546 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556becbd9370 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe85ffff640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbd8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe86ccfe640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe860009690 tx=0x7fe860021950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93bcd310d4991900 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe8477fe640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe8477fe640 1 -- 172.21.15.64:0/2731366546 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86001d040 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe8477fe640 1 -- 172.21.15.64:0/2731366546 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe86002a440 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe858005500 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe8477fe640 1 -- 172.21.15.64:0/2731366546 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe860038050 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe870b5f740 10 client.24751 fetch_fsmap learned FSMap version 24 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.432+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556becc59d90 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe8477fe640 1 -- 172.21.15.64:0/2731366546 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe858005500 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe8477fe640 1 -- 172.21.15.64:0/2731366546 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fe86002abb0 con 0x556becbde810 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe870b5f740 10 client.24751 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe870b5f740 10 client.24751 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe870b5f740 20 client.24751 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:03.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becc5e810 0x556becc5d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:03.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe870b5f740 4 client.24751 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:33:03.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 --> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] -- command(tid 0: {"prefix": "status"}) -- 0x556becb40190 con 0x556becc5e810 2024-08-19T01:33:03.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.433+0000 7fe867fff640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becc5e810 0x556becc5d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe8477fe640 10 client.24751 ms_handle_connect on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe867fff640 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becc5e810 0x556becc5d520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9cdf1b6183acc63 server_cookie=956ff5c77d2486ff in_seq=0 out_seq=0 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe8477fe640 1 -- 172.21.15.64:0/2731366546 <== mds.2 v2:172.21.15.161:6838/2198702470 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556becb40190 con 0x556becc5e810 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe8477fe640 10 client.24751 handle_command_reply: tid=0 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 client.24751 shutdown 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 msgr2=0x556becbd8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe860009690 tx=0x7fe860021950 comp rx=0 tx=0).stop 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 shutdown_connections 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x556becc5e810 0x556becc5d520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556becbde810 0x556becbd8b40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556becbd3f50 0x556becbd8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe8477fe640 0 client.24751 ms_handle_reset on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 --2- 172.21.15.64:0/2731366546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556becbd3630 0x556becbf9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 >> 172.21.15.64:0/2731366546 conn(0x556becbcfbc0 msgr2=0x556becbda7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 shutdown_connections 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 1 -- 172.21.15.64:0/2731366546 wait complete. 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.434+0000 7fe870b5f740 20 client.24751 trim_cache size 0 max 16384 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2024-08-19T01:33:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24436, 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 90.691135582000001, 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 24, 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 93, 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 90, 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 150.898713687 2024-08-19T01:33:03.443 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:33:03.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8540a2c10 msgr2=0x7fe8540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8540a2c10 0x7fe8540a50d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe860020000 tx=0x7fe86001e000 comp rx=0 tx=0).stop 2024-08-19T01:33:03.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8681022a0 msgr2=0x7fe868198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:03.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8681022a0 0x7fe868198740 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe85000c920 tx=0x7fe85000cde0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 shutdown_connections 2024-08-19T01:33:03.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fe8540a2c10 0x7fe8540a50d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe868102c20 0x7fe86819cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8681022a0 0x7fe868198740 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 --2- 172.21.15.64:0/3128699898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe868100b50 0x7fe868198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:03.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 >> 172.21.15.64:0/3128699898 conn(0x7fe8680fc500 msgr2=0x7fe868069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:03.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 shutdown_connections 2024-08-19T01:33:03.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:03.435+0000 7fe86e761640 1 -- 172.21.15.64:0/3128699898 wait complete. 2024-08-19T01:33:03.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:03 smithi064 ceph-mon[18515]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:33:03.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:03 smithi123 ceph-mon[21829]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:33:03.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:03 smithi161 ceph-mon[21757]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:33:04.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.h status 2024-08-19T01:33:04.492 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:04.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 -- 172.21.15.64:0/2375089658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c20101150 msgr2=0x7f2c2010d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:04.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 --2- 172.21.15.64:0/2375089658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c20101150 0x7f2c2010d680 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009cd0 tx=0x7f2c1802a8e0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 -- 172.21.15.64:0/2375089658 shutdown_connections 2024-08-19T01:33:04.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 --2- 172.21.15.64:0/2375089658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c20101150 0x7f2c2010d680 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 --2- 172.21.15.64:0/2375089658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c201007b0 0x7f2c20100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 --2- 172.21.15.64:0/2375089658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c201067b0 0x7f2c20106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 -- 172.21.15.64:0/2375089658 >> 172.21.15.64:0/2375089658 conn(0x7f2c200fc4d0 msgr2=0x7f2c200fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 -- 172.21.15.64:0/2375089658 shutdown_connections 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.968+0000 7f2c27e68640 1 -- 172.21.15.64:0/2375089658 wait complete. 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 Processor -- start 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 -- start start 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c201007b0 0x7f2c201970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c20101150 0x7f2c20197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c201067b0 0x7f2c2019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2c20110bf0 con 0x7f2c201007b0 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2c20110a70 con 0x7f2c20101150 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c27e68640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2c20110d70 con 0x7f2c201067b0 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c25c04640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c201007b0 0x7f2c201970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c25c04640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c201007b0 0x7f2c201970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44192/0 (socket says 172.21.15.64:44192) 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.969+0000 7f2c25c04640 1 -- 172.21.15.64:0/3643650045 learned_addr learned my addr 172.21.15.64:0/3643650045 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c26405640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c201067b0 0x7f2c2019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25403640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c20101150 0x7f2c20197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25c04640 1 -- 172.21.15.64:0/3643650045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c201067b0 msgr2=0x7f2c2019b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25c04640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c201067b0 0x7f2c2019b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25c04640 1 -- 172.21.15.64:0/3643650045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c20101150 msgr2=0x7f2c20197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25c04640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c20101150 0x7f2c20197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25c04640 1 -- 172.21.15.64:0/3643650045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2019c080 con 0x7f2c201007b0 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25403640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c20101150 0x7f2c20197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c25c04640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c201007b0 0x7f2c201970d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0078f0 tx=0x7f2c1c007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c16ffd640 1 -- 172.21.15.64:0/3643650045 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c019020 con 0x7f2c201007b0 2024-08-19T01:33:04.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c2019c370 con 0x7f2c201007b0 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c16ffd640 1 -- 172.21.15.64:0/3643650045 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c1c005500 con 0x7f2c201007b0 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c16ffd640 1 -- 172.21.15.64:0/3643650045 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c00e890 con 0x7f2c201007b0 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.970+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2006a4c0 con 0x7f2c201007b0 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.973+0000 7f2c16ffd640 1 -- 172.21.15.64:0/3643650045 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2c1c0046c0 con 0x7f2c201007b0 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.973+0000 7f2c2a266740 1 Processor -- start 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.973+0000 7f2c2a266740 1 -- start start 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2df2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2e1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c2a266740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55680a239710 con 0x55680a2d4f50 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c2a266740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55680a1d1770 con 0x55680a2d4630 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c2a266740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55680a2356d0 con 0x55680a2df810 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c16ffd640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2c0c0a2b80 0x7f2c0c0a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2df2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2df2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44196/0 (socket says 172.21.15.64:44196) 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 -- 172.21.15.64:0/2465350174 learned_addr learned my addr 172.21.15.64:0/2465350174 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c26405640 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25403640 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2e1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c26405640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2c0c0a2b80 0x7f2c0c0a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c16ffd640 1 -- 172.21.15.64:0/3643650045 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2c1c0d0ae0 con 0x7f2c201007b0 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 -- 172.21.15.64:0/2465350174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 msgr2=0x55680a2e1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2e1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 -- 172.21.15.64:0/2465350174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 msgr2=0x55680a2d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.974+0000 7f2c25c04640 1 -- 172.21.15.64:0/2465350174 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55680a1d83e0 con 0x55680a2d4f50 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.975+0000 7f2c25c04640 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2df2d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00cfd0 tx=0x7f2c1c004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad25e764f26050fe server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:04.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.975+0000 7f2c26405640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2c0c0a2b80 0x7f2c0c0a5040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009430 tx=0x7f2c180095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.975+0000 7f2c0bfff640 1 -- 172.21.15.64:0/2465350174 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c09d700 con 0x55680a2d4f50 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.975+0000 7f2c0bfff640 1 -- 172.21.15.64:0/2465350174 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c1c09d000 con 0x55680a2d4f50 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.975+0000 7f2c0bfff640 1 -- 172.21.15.64:0/2465350174 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c0bbbb0 con 0x55680a2d4f50 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.975+0000 7f2c2a266740 1 -- 172.21.15.64:0/2465350174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 msgr2=0x55680a2df2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.975+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2df2d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00cfd0 tx=0x7f2c1c004bb0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- 172.21.15.64:0/2465350174 shutdown_connections 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2e1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2df2d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2465350174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- 172.21.15.64:0/2465350174 >> 172.21.15.64:0/2465350174 conn(0x55680a2d0bc0 msgr2=0x55680a2d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- 172.21.15.64:0/2465350174 shutdown_connections 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- 172.21.15.64:0/2465350174 wait complete. 2024-08-19T01:33:04.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- start start 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 Processor -- start 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4f50 0x55680a2dcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2df810 0x55680a2dd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55680a2e2380 con 0x55680a2df810 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55680a1f4970 con 0x55680a2d4f50 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55680a2e2500 con 0x55680a2d4630 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.976+0000 7f2c2a266740 10 client.0 _ll_register_callbacks cb 0x556809f487b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2df810 0x55680a2dd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2df810 0x55680a2dd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44206/0 (socket says 172.21.15.64:44206) 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 -- 172.21.15.64:0/2656851073 learned_addr learned my addr 172.21.15.64:0/2656851073 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c25c04640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c25403640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4f50 0x55680a2dcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 -- 172.21.15.64:0/2656851073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 msgr2=0x55680a2dc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2dc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 -- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4f50 msgr2=0x55680a2dcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4f50 0x55680a2dcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 -- 172.21.15.64:0/2656851073 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556809fab2b0 con 0x55680a2df810 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c25403640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4f50 0x55680a2dcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c26405640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2df810 0x55680a2dd220 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2c1802aff0 tx=0x7f2c1802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29951e1153a926d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c097fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c097fa640 1 -- 172.21.15.64:0/2656851073 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1804f030 con 0x55680a2df810 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c097fa640 1 -- 172.21.15.64:0/2656851073 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2c18004250 con 0x55680a2df810 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c1c09d000 con 0x55680a2df810 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c097fa640 1 -- 172.21.15.64:0/2656851073 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2c1805c050 con 0x55680a2df810 2024-08-19T01:33:04.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c2a266740 10 client.15210 fetch_fsmap learned FSMap version 24 2024-08-19T01:33:04.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.977+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55680a2ee070 con 0x55680a2df810 2024-08-19T01:33:04.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c097fa640 1 -- 172.21.15.64:0/2656851073 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c1c09d000 con 0x55680a2df810 2024-08-19T01:33:04.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c097fa640 1 -- 172.21.15.64:0/2656851073 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f2c18030070 con 0x55680a2df810 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c2a266740 10 client.15210 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c2a266740 10 client.15210 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c2a266740 20 client.15210 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a2f06b0 0x55680a2f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c2a266740 4 client.15210 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 --> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55680a246c80 con 0x55680a2f06b0 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c25c04640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a2f06b0 0x55680a2f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c097fa640 10 client.15210 ms_handle_connect on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:04.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.978+0000 7f2c25c04640 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a2f06b0 0x55680a2f1b20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb7f978b9d71920d server_cookie=d60473412446ffce in_seq=0 out_seq=0 2024-08-19T01:33:04.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c097fa640 1 -- 172.21.15.64:0/2656851073 <== mds.1 v2:172.21.15.123:6836/3425782983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55680a246c80 con 0x55680a2f06b0 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c097fa640 10 client.15210 handle_command_reply: tid=0 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 client.15210 shutdown 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2df810 msgr2=0x55680a2dd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2df810 0x55680a2dd220 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2c1802aff0 tx=0x7f2c1802a8e0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 shutdown_connections 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a2f06b0 0x55680a2f1b20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2df810 0x55680a2dd220 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4f50 0x55680a2dcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c097fa640 0 client.15210 ms_handle_reset on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2656851073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2dc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.979+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 >> 172.21.15.64:0/2656851073 conn(0x55680a2d0bc0 msgr2=0x55680a2cfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.980+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 shutdown_connections 2024-08-19T01:33:04.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.980+0000 7f2c2a266740 1 -- 172.21.15.64:0/2656851073 wait complete. 2024-08-19T01:33:04.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.980+0000 7f2c2a266740 20 client.15210 trim_cache size 0 max 16384 2024-08-19T01:33:05.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 Processor -- start 2024-08-19T01:33:05.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 -- start start 2024-08-19T01:33:05.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:05.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2d7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:05.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2db780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55680a2efe60 con 0x55680a2d4f50 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55680a2f0530 con 0x55680a2d4630 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c2a266740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55680a2f0c00 con 0x55680a2df810 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2d7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2d7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44222/0 (socket says 172.21.15.64:44222) 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 -- 172.21.15.64:0/2976968492 learned_addr learned my addr 172.21.15.64:0/2976968492 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25c04640 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c26405640 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2db780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 -- 172.21.15.64:0/2976968492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 msgr2=0x55680a2db780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2db780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 -- 172.21.15.64:0/2976968492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 msgr2=0x55680a2d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 -- 172.21.15.64:0/2976968492 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55680a2d4460 con 0x55680a2d4f50 2024-08-19T01:33:05.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c26405640 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2db780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:05.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.998+0000 7f2c25403640 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2d7af0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000a9f0 tx=0x7f2c1000aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d180afd76e5b8 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:05.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c0b7fe640 1 -- 172.21.15.64:0/2976968492 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1001a070 con 0x55680a2d4f50 2024-08-19T01:33:05.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c0b7fe640 1 -- 172.21.15.64:0/2976968492 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c10012de0 con 0x55680a2d4f50 2024-08-19T01:33:05.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c0b7fe640 1 -- 172.21.15.64:0/2976968492 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c100148b0 con 0x55680a2d4f50 2024-08-19T01:33:05.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 -- 172.21.15.64:0/2976968492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 msgr2=0x55680a2d7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:05.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2d7af0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000a9f0 tx=0x7f2c1000aeb0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 -- 172.21.15.64:0/2976968492 shutdown_connections 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2df810 0x55680a2db780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2d7af0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 --2- 172.21.15.64:0/2976968492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2d4630 0x55680a2d0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 -- 172.21.15.64:0/2976968492 >> 172.21.15.64:0/2976968492 conn(0x55680a2d0bc0 msgr2=0x55680a2db070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 -- 172.21.15.64:0/2976968492 shutdown_connections 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:04.999+0000 7f2c2a266740 1 -- 172.21.15.64:0/2976968492 wait complete. 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 -- start start 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 Processor -- start 2024-08-19T01:33:05.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2da1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2df810 0x55680a2d8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55680a2f19a0 con 0x55680a2d4f50 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55680a2f12d0 con 0x55680a2df810 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55680a2f2070 con 0x55680a2d4630 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c2a266740 10 client.0 _ll_register_callbacks cb 0x7f2c18031790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c25403640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2da1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c25403640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2da1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44232/0 (socket says 172.21.15.64:44232) 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c25403640 1 -- 172.21.15.64:0/537866602 learned_addr learned my addr 172.21.15.64:0/537866602 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.000+0000 7f2c26405640 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2df810 0x55680a2d8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c25c04640 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c25403640 1 -- 172.21.15.64:0/537866602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 msgr2=0x55680a2d9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c25403640 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2d9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c25403640 1 -- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2df810 msgr2=0x55680a2d8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:05.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c25403640 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2df810 0x55680a2d8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c25403640 1 -- 172.21.15.64:0/537866602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55680a2cc130 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c25403640 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2da1f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2c10008010 tx=0x7f2c10015b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc3860c4820a97d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c097fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c097fa640 1 -- 172.21.15.64:0/537866602 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1002c060 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c097fa640 1 -- 172.21.15.64:0/537866602 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2c10012be0 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c10012de0 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c097fa640 1 -- 172.21.15.64:0/537866602 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2c10033050 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c2a266740 10 client.15216 fetch_fsmap learned FSMap version 24 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55680a35f640 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c097fa640 1 -- 172.21.15.64:0/537866602 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c10012de0 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.001+0000 7f2c097fa640 1 -- 172.21.15.64:0/537866602 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f2c1001f5e0 con 0x55680a2d4f50 2024-08-19T01:33:05.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 10 client.15216 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 10 client.15216 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 20 client.15216 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a363ea0 0x55680a362bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 4 client.15216 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 --> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] -- command(tid 0: {"prefix": "status"}) -- 0x55680a241190 con 0x55680a363ea0 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c25c04640 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a363ea0 0x55680a362bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c25c04640 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a363ea0 0x55680a362bb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aab144d50d79a0d9 server_cookie=6ff93025e1ad1816 in_seq=0 out_seq=0 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c097fa640 10 client.15216 ms_handle_connect on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c097fa640 1 -- 172.21.15.64:0/537866602 <== mds.1 v2:172.21.15.123:6836/3425782983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55680a241190 con 0x55680a363ea0 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c097fa640 10 client.15216 handle_command_reply: tid=0 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 1 client.15216 shutdown 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 msgr2=0x55680a2da1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.002+0000 7f2c2a266740 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2da1f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2c10008010 tx=0x7f2c10015b60 comp rx=0 tx=0).stop 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 shutdown_connections 2024-08-19T01:33:05.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x55680a363ea0 0x55680a362bb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55680a2df810 0x55680a2d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55680a2d4f50 0x55680a2da1f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c097fa640 0 client.15216 ms_handle_reset on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 --2- 172.21.15.64:0/537866602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55680a2d4630 0x55680a2d9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 >> 172.21.15.64:0/537866602 conn(0x55680a2d0bc0 msgr2=0x55680a2db890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 shutdown_connections 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 1 -- 172.21.15.64:0/537866602 wait complete. 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c2a266740 20 client.15216 trim_cache size 0 max 16384 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:33:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24452, 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 93.264338132000006, 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 24, 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 93, 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 90, 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 155.08564958599999 2024-08-19T01:33:05.011 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2c0c0a2b80 msgr2=0x7f2c0c0a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2c0c0a2b80 0x7f2c0c0a5040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009430 tx=0x7f2c180095c0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c201007b0 msgr2=0x7f2c201970d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c201007b0 0x7f2c201970d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0078f0 tx=0x7f2c1c007db0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 shutdown_connections 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f2c0c0a2b80 0x7f2c0c0a5040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c201067b0 0x7f2c2019b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c20101150 0x7f2c20197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.003+0000 7f2c27e68640 1 --2- 172.21.15.64:0/3643650045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c201007b0 0x7f2c201970d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:05.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.004+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 >> 172.21.15.64:0/3643650045 conn(0x7f2c200fc4d0 msgr2=0x7f2c200fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:05.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.004+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 shutdown_connections 2024-08-19T01:33:05.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:05.004+0000 7f2c27e68640 1 -- 172.21.15.64:0/3643650045 wait complete. 2024-08-19T01:33:05.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.b status 2024-08-19T01:33:05.898 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:05 smithi064 ceph-mon[18515]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:05.898 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:05 smithi064 ceph-mon[18515]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-08-19T01:33:05.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:05 smithi123 ceph-mon[21829]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:05.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:05 smithi123 ceph-mon[21829]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-08-19T01:33:05.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:05 smithi161 ceph-mon[21757]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:05 smithi161 ceph-mon[21757]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-08-19T01:33:06.133 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:06.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 -- 172.21.15.64:0/3649016493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 msgr2=0x7f0504100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 --2- 172.21.15.64:0/3649016493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504100c20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009930 tx=0x7f04f402a420 comp rx=0 tx=0).stop 2024-08-19T01:33:06.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 -- 172.21.15.64:0/3649016493 shutdown_connections 2024-08-19T01:33:06.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 --2- 172.21.15.64:0/3649016493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0504101160 0x7f050410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 --2- 172.21.15.64:0/3649016493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504100c20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 --2- 172.21.15.64:0/3649016493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05041067c0 0x7f0504106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 -- 172.21.15.64:0/3649016493 >> 172.21.15.64:0/3649016493 conn(0x7f05040fc500 msgr2=0x7f05040fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 -- 172.21.15.64:0/3649016493 shutdown_connections 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.585+0000 7f050b83e640 1 -- 172.21.15.64:0/3649016493 wait complete. 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 Processor -- start 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 -- start start 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0504101160 0x7f0504199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05041067c0 0x7f05041a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0504110be0 con 0x7f05041007c0 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0504110a60 con 0x7f0504101160 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f050b83e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0504110d60 con 0x7f05041067c0 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f05095da640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f05095da640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44250/0 (socket says 172.21.15.64:44250) 2024-08-19T01:33:06.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f05095da640 1 -- 172.21.15.64:0/3910549141 learned_addr learned my addr 172.21.15.64:0/3910549141 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.586+0000 7f0508dd9640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0504101160 0x7f0504199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f0509ddb640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05041067c0 0x7f05041a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f05095da640 1 -- 172.21.15.64:0/3910549141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05041067c0 msgr2=0x7f05041a40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f05095da640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05041067c0 0x7f05041a40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f05095da640 1 -- 172.21.15.64:0/3910549141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0504101160 msgr2=0x7f0504199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f05095da640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0504101160 0x7f0504199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f05095da640 1 -- 172.21.15.64:0/3910549141 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05041a4610 con 0x7f05041007c0 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f0508dd9640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0504101160 0x7f0504199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f0509ddb640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05041067c0 0x7f05041a40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f05095da640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504199050 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f050000b4a0 tx=0x7f050000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f04fa7fc640 1 -- 172.21.15.64:0/3910549141 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500014070 con 0x7f05041007c0 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f04fa7fc640 1 -- 172.21.15.64:0/3910549141 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0500010af0 con 0x7f05041007c0 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05041a47e0 con 0x7f05041007c0 2024-08-19T01:33:06.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f04fa7fc640 1 -- 172.21.15.64:0/3910549141 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500017d10 con 0x7f05041007c0 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.587+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05041a4ca0 con 0x7f05041007c0 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.589+0000 7f04fa7fc640 1 -- 172.21.15.64:0/3910549141 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0500017070 con 0x7f05041007c0 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 Processor -- start 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 -- start start 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0d630 0x563dd6f0da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f182d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f18810 0x563dd6f1ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563dd6e72710 con 0x563dd6f0d630 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563dd6e0a770 con 0x563dd6f18810 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f050dc3c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563dd6e6e6d0 con 0x563dd6f0df50 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f04fa7fc640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04f00a2b80 0x7f04f00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0509ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0d630 0x563dd6f0da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0509ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0d630 0x563dd6f0da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44260/0 (socket says 172.21.15.64:44260) 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0509ddb640 1 -- 172.21.15.64:0/1695511573 learned_addr learned my addr 172.21.15.64:0/1695511573 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f04fa7fc640 1 -- 172.21.15.64:0/3910549141 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f05000ce890 con 0x7f05041007c0 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f05095da640 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f182d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0508dd9640 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f18810 0x563dd6f1ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0509ddb640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04f00a2b80 0x7f04f00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0509ddb640 1 -- 172.21.15.64:0/1695511573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 msgr2=0x563dd6f182d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0509ddb640 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f182d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.590+0000 7f0509ddb640 1 -- 172.21.15.64:0/1695511573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f18810 msgr2=0x563dd6f1ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.591+0000 7f0509ddb640 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f18810 0x563dd6f1ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.591+0000 7f0509ddb640 1 -- 172.21.15.64:0/1695511573 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd6e113e0 con 0x563dd6f0d630 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.591+0000 7f0509ddb640 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0d630 0x563dd6f0da10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0149c0 tx=0x7f04fc014e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb23ea92596e892e server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.591+0000 7f04ef7fe640 1 -- 172.21.15.64:0/1695511573 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc021070 con 0x563dd6f0d630 2024-08-19T01:33:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.591+0000 7f04ef7fe640 1 -- 172.21.15.64:0/1695511573 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04fc013460 con 0x563dd6f0d630 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.591+0000 7f04ef7fe640 1 -- 172.21.15.64:0/1695511573 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00f630 con 0x563dd6f0d630 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 -- 172.21.15.64:0/1695511573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0d630 msgr2=0x563dd6f0da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0d630 0x563dd6f0da10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0149c0 tx=0x7f04fc014e80 comp rx=0 tx=0).stop 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f0509ddb640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04f00a2b80 0x7f04f00a5040 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f04fc01e000 tx=0x7f04fc01a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 -- 172.21.15.64:0/1695511573 shutdown_connections 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f18810 0x563dd6f1ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f182d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 --2- 172.21.15.64:0/1695511573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0d630 0x563dd6f0da10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 -- 172.21.15.64:0/1695511573 >> 172.21.15.64:0/1695511573 conn(0x563dd6f09bc0 msgr2=0x563dd6f09fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 -- 172.21.15.64:0/1695511573 shutdown_connections 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 -- 172.21.15.64:0/1695511573 wait complete. 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.592+0000 7f050dc3c740 1 -- start start 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 1 Processor -- start 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f15700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f15c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f16180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f1b3e0 con 0x563dd6f18810 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563dd6e2d970 con 0x563dd6f0d630 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f1b560 con 0x563dd6f0df50 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f050dc3c740 10 client.0 _ll_register_callbacks cb 0x563dd6b817b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f05095da640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f15700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f0509ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f16180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f0508dd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f15c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.593+0000 7f0509ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f16180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44262/0 (socket says 172.21.15.64:44262) 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f0509ddb640 1 -- 172.21.15.64:0/194162721 learned_addr learned my addr 172.21.15.64:0/194162721 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f0509ddb640 1 -- 172.21.15.64:0/194162721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 msgr2=0x563dd6f15c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f0509ddb640 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f15c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f0509ddb640 1 -- 172.21.15.64:0/194162721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 msgr2=0x563dd6f15700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f0509ddb640 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f15700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f0509ddb640 1 -- 172.21.15.64:0/194162721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd6be42b0 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f0509ddb640 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f16180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f04fc013bb0 tx=0x7f04fc008230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a0a1bd3d331224 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f04ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f04ecff9640 1 -- 172.21.15.64:0/194162721 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc021070 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f04ecff9640 1 -- 172.21.15.64:0/194162721 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f04fc013fc0 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.594+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04fc013460 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f04ecff9640 1 -- 172.21.15.64:0/194162721 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f04fc039050 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 10 client.15228 fetch_fsmap learned FSMap version 24 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563dd6f27090 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f04ecff9640 1 -- 172.21.15.64:0/194162721 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04fc013460 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f04ecff9640 1 -- 172.21.15.64:0/194162721 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f04fc02e020 con 0x563dd6f18810 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 10 client.15228 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:33:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 10 client.15228 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 20 client.15228 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f29530 0x563dd6f2a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 4 client.15228 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dd6e7fc80 con 0x563dd6f29530 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.595+0000 7f05095da640 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f29530 0x563dd6f2a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.596+0000 7f05095da640 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f29530 0x563dd6f2a9a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3df0ec27b07d7e66 server_cookie=eac27c0461b1e9be in_seq=0 out_seq=0 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.596+0000 7f04ecff9640 10 client.15228 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f04ecff9640 1 -- 172.21.15.64:0/194162721 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563dd6e7fc80 con 0x563dd6f29530 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f04ecff9640 10 client.15228 handle_command_reply: tid=0 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 client.15228 shutdown 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 msgr2=0x563dd6f16180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f16180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f04fc013bb0 tx=0x7f04fc008230 comp rx=0 tx=0).stop 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 shutdown_connections 2024-08-19T01:33:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f16180 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f15c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f29530 0x563dd6f2a9a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 --2- 172.21.15.64:0/194162721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f15700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 >> 172.21.15.64:0/194162721 conn(0x563dd6f09bc0 msgr2=0x563dd6f08d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f04ecff9640 0 client.15228 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 shutdown_connections 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 1 -- 172.21.15.64:0/194162721 wait complete. 2024-08-19T01:33:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.597+0000 7f050dc3c740 20 client.15228 trim_cache size 0 max 16384 2024-08-19T01:33:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 Processor -- start 2024-08-19T01:33:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 -- start start 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f10090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0df50 0x563dd6f15c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f18810 0x563dd6f16190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f28ce0 con 0x563dd6f0df50 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f293b0 con 0x563dd6f0d630 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f050dc3c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f29a80 con 0x563dd6f18810 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0df50 0x563dd6f15c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0df50 0x563dd6f15c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44264/0 (socket says 172.21.15.64:44264) 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 -- 172.21.15.64:0/2202007221 learned_addr learned my addr 172.21.15.64:0/2202007221 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0509ddb640 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f18810 0x563dd6f16190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f05095da640 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f10090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 -- 172.21.15.64:0/2202007221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f18810 msgr2=0x563dd6f16190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f18810 0x563dd6f16190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 -- 172.21.15.64:0/2202007221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 msgr2=0x563dd6f10090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f10090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 -- 172.21.15.64:0/2202007221 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd6f05130 con 0x563dd6f0df50 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0509ddb640 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f18810 0x563dd6f16190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.616+0000 7f0508dd9640 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0df50 0x563dd6f15c50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009930 tx=0x7f04f4031880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bab207d114fa09 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f04eeffd640 1 -- 172.21.15.64:0/2202007221 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4035040 con 0x563dd6f0df50 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f04eeffd640 1 -- 172.21.15.64:0/2202007221 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f40035f0 con 0x563dd6f0df50 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f04eeffd640 1 -- 172.21.15.64:0/2202007221 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4031a30 con 0x563dd6f0df50 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 -- 172.21.15.64:0/2202007221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0df50 msgr2=0x563dd6f15c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0df50 0x563dd6f15c50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009930 tx=0x7f04f4031880 comp rx=0 tx=0).stop 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 -- 172.21.15.64:0/2202007221 shutdown_connections 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f18810 0x563dd6f16190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f0df50 0x563dd6f15c50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2202007221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f10090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 -- 172.21.15.64:0/2202007221 >> 172.21.15.64:0/2202007221 conn(0x563dd6f09bc0 msgr2=0x563dd6f09fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 -- 172.21.15.64:0/2202007221 shutdown_connections 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.617+0000 7f050dc3c740 1 -- 172.21.15.64:0/2202007221 wait complete. 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 -- start start 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 Processor -- start 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f32d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f332b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f12160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f2a820 con 0x563dd6f18810 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f2a150 con 0x563dd6f0d630 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563dd6f2aef0 con 0x563dd6f0df50 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f050dc3c740 10 client.0 _ll_register_callbacks cb 0x7f04fc01f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f0509ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f12160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f0509ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f12160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44272/0 (socket says 172.21.15.64:44272) 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f0509ddb640 1 -- 172.21.15.64:0/2407372617 learned_addr learned my addr 172.21.15.64:0/2407372617 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.618+0000 7f05095da640 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f32d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f0508dd9640 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f0509ddb640 1 -- 172.21.15.64:0/2407372617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 msgr2=0x563dd6f332b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f0509ddb640 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f332b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f0509ddb640 1 -- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 msgr2=0x563dd6f32d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f0509ddb640 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f32d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f0509ddb640 1 -- 172.21.15.64:0/2407372617 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd6f12990 con 0x563dd6f18810 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f0509ddb640 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f12160 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f04fc005400 tx=0x7f04fc004ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea282056382b6c1 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f04ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f04ecff9640 1 -- 172.21.15.64:0/2407372617 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc021040 con 0x563dd6f18810 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f04ecff9640 1 -- 172.21.15.64:0/2407372617 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f04fc013bf0 con 0x563dd6f18810 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04f40035f0 con 0x563dd6f18810 2024-08-19T01:33:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f04ecff9640 1 -- 172.21.15.64:0/2407372617 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f04fc047050 con 0x563dd6f18810 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f050dc3c740 10 client.15234 fetch_fsmap learned FSMap version 24 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563dd6f31380 con 0x563dd6f18810 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.619+0000 7f04ecff9640 1 -- 172.21.15.64:0/2407372617 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f40035f0 con 0x563dd6f18810 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f04ecff9640 1 -- 172.21.15.64:0/2407372617 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f04fc03c020 con 0x563dd6f18810 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f050dc3c740 10 client.15234 fetch_fsmap finished waiting for FSMap version 24 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f050dc3c740 10 client.15234 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f050dc3c740 20 client.15234 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f98220 0x563dd6f96f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f050dc3c740 4 client.15234 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "status"}) -- 0x563dd6e7a190 con 0x563dd6f98220 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f05095da640 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f98220 0x563dd6f96f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f05095da640 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f98220 0x563dd6f96f30 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fd96ed678943d03 server_cookie=6ba9892ee039c0c in_seq=0 out_seq=0 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.620+0000 7f04ecff9640 10 client.15234 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f04ecff9640 1 -- 172.21.15.64:0/2407372617 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563dd6e7a190 con 0x563dd6f98220 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f04ecff9640 10 client.15234 handle_command_reply: tid=0 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 client.15234 shutdown 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 msgr2=0x563dd6f12160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f12160 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f04fc005400 tx=0x7f04fc004ec0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 shutdown_connections 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563dd6f18810 0x563dd6f12160 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dd6f0df50 0x563dd6f332b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x563dd6f98220 0x563dd6f96f30 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f04ecff9640 0 client.15234 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 --2- 172.21.15.64:0/2407372617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563dd6f0d630 0x563dd6f32d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 >> 172.21.15.64:0/2407372617 conn(0x563dd6f09bc0 msgr2=0x563dd6f08520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 shutdown_connections 2024-08-19T01:33:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 1 -- 172.21.15.64:0/2407372617 wait complete. 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.621+0000 7f050dc3c740 20 client.15234 trim_cache size 0 max 16384 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24476, 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 134.07492951899999, 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 24, 2024-08-19T01:33:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 93, 2024-08-19T01:33:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 90, 2024-08-19T01:33:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 145.08199142199999 2024-08-19T01:33:06.636 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:33:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04f00a2b80 msgr2=0x7f04f00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04f00a2b80 0x7f04f00a5040 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f04fc01e000 tx=0x7f04fc01a000 comp rx=0 tx=0).stop 2024-08-19T01:33:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 msgr2=0x7f0504199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504199050 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f050000b4a0 tx=0x7f050000b960 comp rx=0 tx=0).stop 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 shutdown_connections 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f04f00a2b80 0x7f04f00a5040 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05041067c0 0x7f05041a40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0504101160 0x7f0504199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 --2- 172.21.15.64:0/3910549141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05041007c0 0x7f0504199050 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 >> 172.21.15.64:0/3910549141 conn(0x7f05040fc500 msgr2=0x7f05040fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 shutdown_connections 2024-08-19T01:33:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:06.622+0000 7f050b83e640 1 -- 172.21.15.64:0/3910549141 wait complete. 2024-08-19T01:33:07.432 INFO:teuthology.run_tasks:Running task sequential... 2024-08-19T01:33:07.443 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-08-19T01:33:07.443 INFO:teuthology.task.sequential:In sequential, running task print... 2024-08-19T01:33:07.443 INFO:teuthology.task.print:Enabling mgr modules 2024-08-19T01:33:07.443 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-08-19T01:33:07.444 INFO:teuthology.task.exec:Executing custom commands... 2024-08-19T01:33:07.444 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:33:07.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-08-19T01:33:07.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 -- 172.21.15.64:0/3974162352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 msgr2=0x7f0240108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:07.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 --2- 172.21.15.64:0/3974162352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f0240108bb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0238009ca0 tx=0x7f023802a8c0 comp rx=0 tx=0).stop 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 -- 172.21.15.64:0/3974162352 shutdown_connections 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 --2- 172.21.15.64:0/3974162352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f0240108bb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 --2- 172.21.15.64:0/3974162352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0240101960 0x7f0240101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 --2- 172.21.15.64:0/3974162352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0240100770 0x7f0240100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 -- 172.21.15.64:0/3974162352 >> 172.21.15.64:0/3974162352 conn(0x7f02400fbf10 msgr2=0x7f02400fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 -- 172.21.15.64:0/3974162352 shutdown_connections 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.576+0000 7f0246b91640 1 -- 172.21.15.64:0/3974162352 wait complete. 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 Processor -- start 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 -- start start 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0240100770 0x7f0240197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0240101960 0x7f0240198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f024019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f024010d580 con 0x7f0240102320 2024-08-19T01:33:07.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f024010d400 con 0x7f0240100770 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f0246b91640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f024010d700 con 0x7f0240101960 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f024019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f024019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44298/0 (socket says 172.21.15.64:44298) 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 -- 172.21.15.64:0/3040643074 learned_addr learned my addr 172.21.15.64:0/3040643074 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024492d640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0240100770 0x7f0240197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f023ffff640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0240101960 0x7f0240198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 -- 172.21.15.64:0/3040643074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0240101960 msgr2=0x7f0240198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0240101960 0x7f0240198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 -- 172.21.15.64:0/3040643074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0240100770 msgr2=0x7f0240197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0240100770 0x7f0240197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 -- 172.21.15.64:0/3040643074 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024019f770 con 0x7f0240102320 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f023ffff640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0240101960 0x7f0240198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:07.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.577+0000 7f024512e640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f024019f100 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0238005e30 tx=0x7f0238031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:07.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.578+0000 7f023dffb640 1 -- 172.21.15.64:0/3040643074 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238003b70 con 0x7f0240102320 2024-08-19T01:33:07.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.578+0000 7f023dffb640 1 -- 172.21.15.64:0/3040643074 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0238003d10 con 0x7f0240102320 2024-08-19T01:33:07.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.578+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024019fa00 con 0x7f0240102320 2024-08-19T01:33:07.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.578+0000 7f023dffb640 1 -- 172.21.15.64:0/3040643074 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023802fd30 con 0x7f0240102320 2024-08-19T01:33:07.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.578+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024019ffa0 con 0x7f0240102320 2024-08-19T01:33:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.579+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0240107a90 con 0x7f0240102320 2024-08-19T01:33:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.580+0000 7f023dffb640 1 -- 172.21.15.64:0/3040643074 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0238004050 con 0x7f0240102320 2024-08-19T01:33:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.581+0000 7f023dffb640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f02300a2cb0 0x7f02300a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.581+0000 7f024492d640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f02300a2cb0 0x7f02300a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.582+0000 7f023dffb640 1 -- 172.21.15.64:0/3040643074 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f02380f0890 con 0x7f0240102320 2024-08-19T01:33:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.582+0000 7f024492d640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f02300a2cb0 0x7f02300a5170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02401017c0 tx=0x7f0234009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:07.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.585+0000 7f023dffb640 1 -- 172.21.15.64:0/3040643074 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f02380bd3c0 con 0x7f0240102320 2024-08-19T01:33:07.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:07 smithi064 ceph-mon[18515]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-08-19T01:33:07.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:07.809+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f0240062b20 con 0x7f0240102320 2024-08-19T01:33:07.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:07 smithi123 ceph-mon[21829]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-08-19T01:33:07.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:07 smithi161 ceph-mon[21757]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-08-19T01:33:08.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.648+0000 7f023dffb640 1 -- 172.21.15.64:0/3040643074 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f02380bccc0 con 0x7f0240102320 2024-08-19T01:33:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f02300a2cb0 msgr2=0x7f02300a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f02300a2cb0 0x7f02300a5170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02401017c0 tx=0x7f0234009110 comp rx=0 tx=0).stop 2024-08-19T01:33:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 msgr2=0x7f024019f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f024019f100 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0238005e30 tx=0x7f0238031ee0 comp rx=0 tx=0).stop 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 shutdown_connections 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f02300a2cb0 0x7f02300a5170 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0240102320 0x7f024019f100 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0240101960 0x7f0240198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 --2- 172.21.15.64:0/3040643074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0240100770 0x7f0240197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 >> 172.21.15.64:0/3040643074 conn(0x7f02400fbf10 msgr2=0x7f0240104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 shutdown_connections 2024-08-19T01:33:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:08.651+0000 7f0246b91640 1 -- 172.21.15.64:0/3040643074 wait complete. 2024-08-19T01:33:08.671 INFO:teuthology.run_tasks:Running task kclient... 2024-08-19T01:33:08.684 INFO:tasks.kclient:Mounting kernel clients... 2024-08-19T01:33:08.684 INFO:tasks.kclient:config is {'client.0': {}} 2024-08-19T01:33:08.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:08.685 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-08-19T01:33:08.701 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-08-19T01:33:08.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:08.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-08-19T01:33:08.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:08.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link delete ceph-brx 2024-08-19T01:33:08.889 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-08-19T01:33:08.889 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-19T01:33:08.889 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:08.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs ls 2024-08-19T01:33:08.893 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:08 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3040643074' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-08-19T01:33:08.893 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: ignoring --setuser ceph since I am not root 2024-08-19T01:33:08.893 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: ignoring --setgroup ceph since I am not root 2024-08-19T01:33:08.898 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ignoring --setuser ceph since I am not root 2024-08-19T01:33:08.898 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ignoring --setgroup ceph since I am not root 2024-08-19T01:33:08.899 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:08 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3040643074' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-08-19T01:33:08.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:08 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3040643074' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-08-19T01:33:08.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: ignoring --setuser ceph since I am not root 2024-08-19T01:33:08.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: ignoring --setgroup ceph since I am not root 2024-08-19T01:33:08.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:08 smithi123 ceph-mgr[23203]: -- 172.21.15.123:0/1094889075 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a3f93650e0 con 0x55a3f9342800 2024-08-19T01:33:08.917 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:08.892+0000 7f0df5a95040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:33:09.153 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:09.199 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:08.890+0000 7fb2f7816040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:33:09.199 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:08.956+0000 7fb2f7816040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:33:09.199 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:09.114+0000 7fb2f7816040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:33:09.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:08.904+0000 7fd58121c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T01:33:09.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:08.975+0000 7fd58121c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:33:09.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:09.131+0000 7fd58121c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:33:09.261 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:08.959+0000 7f0df5a95040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T01:33:09.261 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:09.102+0000 7f0df5a95040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:33:09.261 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:09.259+0000 7f0df5a95040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:33:09.527 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:09.403+0000 7f0df5a95040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:33:09.545 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:09.275+0000 7fb2f7816040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:33:09.546 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:09.419+0000 7fb2f7816040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:33:09.546 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:09.542+0000 7fb2f7816040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:33:09.576 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:09.303+0000 7fd58121c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T01:33:09.576 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:09.453+0000 7fd58121c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 -- 172.21.15.64:0/3725011137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b58102250 msgr2=0x7f8b5810e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/3725011137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b58102250 0x7f8b5810e780 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8b50009cd0 tx=0x7f8b5002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 -- 172.21.15.64:0/3725011137 shutdown_connections 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/3725011137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b58102250 0x7f8b5810e780 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/3725011137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b581018b0 0x7f8b58101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/3725011137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b581078b0 0x7f8b58107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 -- 172.21.15.64:0/3725011137 >> 172.21.15.64:0/3725011137 conn(0x7f8b580fd5f0 msgr2=0x7f8b580ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 -- 172.21.15.64:0/3725011137 shutdown_connections 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.658+0000 7f8b5ca16640 1 -- 172.21.15.64:0/3725011137 wait complete. 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 Processor -- start 2024-08-19T01:33:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 -- start start 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b581018b0 0x7f8b58198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b58102250 0x7f8b581986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b581078b0 0x7f8b5819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58111cb0 con 0x7f8b581018b0 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58111b30 con 0x7f8b58102250 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5ca16640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58111e30 con 0x7f8b581078b0 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b581018b0 0x7f8b58198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b581018b0 0x7f8b58198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35220/0 (socket says 172.21.15.64:35220) 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.659+0000 7f8b5659c640 1 -- 172.21.15.64:0/1413010870 learned_addr learned my addr 172.21.15.64:0/1413010870 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b55d9b640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b58102250 0x7f8b581986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b56d9d640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b581078b0 0x7f8b5819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5659c640 1 -- 172.21.15.64:0/1413010870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b581078b0 msgr2=0x7f8b5819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5659c640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b581078b0 0x7f8b5819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5659c640 1 -- 172.21.15.64:0/1413010870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b58102250 msgr2=0x7f8b581986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5659c640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b58102250 0x7f8b581986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5659c640 1 -- 172.21.15.64:0/1413010870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5819d140 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5659c640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b581018b0 0x7f8b58198190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800e9f0 tx=0x7f8b4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b477fe640 1 -- 172.21.15.64:0/1413010870 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800cd80 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b5819d430 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b477fe640 1 -- 172.21.15.64:0/1413010870 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b48010910 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b477fe640 1 -- 172.21.15.64:0/1413010870 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48018990 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.660+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b581a4d10 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.661+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b58101d10 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.663+0000 7f8b477fe640 1 -- 172.21.15.64:0/1413010870 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f8b48010ac0 con 0x7f8b581018b0 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.663+0000 7f8b477fe640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 0x7f8b400a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.663+0000 7f8b55d9b640 1 -- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 msgr2=0x7f8b400a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2663741075 2024-08-19T01:33:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.664+0000 7f8b477fe640 1 -- 172.21.15.64:0/1413010870 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8b480cfb60 con 0x7f8b581018b0 2024-08-19T01:33:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.664+0000 7f8b55d9b640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 0x7f8b400a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:33:09.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.666+0000 7f8b477fe640 1 -- 172.21.15.64:0/1413010870 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f8b4809c0a0 con 0x7f8b581018b0 2024-08-19T01:33:09.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.865+0000 7f8b55d9b640 1 -- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 msgr2=0x7f8b400a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2663741075 2024-08-19T01:33:09.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.865+0000 7f8b55d9b640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 0x7f8b400a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-19T01:33:09.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3040643074' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-08-19T01:33:09.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-mon[18515]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-08-19T01:33:09.867 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:09.583+0000 7fd58121c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:33:09.867 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:09.716+0000 7fd58121c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:33:09.867 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:09.776+0000 7fd58121c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:33:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.880+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8b58109e20 con 0x7f8b581018b0 2024-08-19T01:33:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.881+0000 7f8b477fe640 1 -- 172.21.15.64:0/1413010870 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f8b480a03a0 con 0x7f8b581018b0 2024-08-19T01:33:09.882 INFO:teuthology.orchestra.run.smithi064.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-08-19T01:33:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 msgr2=0x7f8b400a5090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:33:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 0x7f8b400a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b581018b0 msgr2=0x7f8b58198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b581018b0 0x7f8b58198190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800e9f0 tx=0x7f8b4800eeb0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 shutdown_connections 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f8b400a2bd0 0x7f8b400a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b581078b0 0x7f8b5819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b58102250 0x7f8b581986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 --2- 172.21.15.64:0/1413010870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b581018b0 0x7f8b58198190 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 >> 172.21.15.64:0/1413010870 conn(0x7f8b580fd5f0 msgr2=0x7f8b580fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 shutdown_connections 2024-08-19T01:33:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:09.883+0000 7f8b5ca16640 1 -- 172.21.15.64:0/1413010870 wait complete. 2024-08-19T01:33:09.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3040643074' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-08-19T01:33:09.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-mon[21829]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-08-19T01:33:09.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:09.526+0000 7f0df5a95040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T01:33:09.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:09.650+0000 7f0df5a95040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:33:09.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:09.708+0000 7f0df5a95040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:33:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3040643074' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-08-19T01:33:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-mon[21757]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-08-19T01:33:09.949 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:09.668+0000 7fb2f7816040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T01:33:09.949 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:09.725+0000 7fb2f7816040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T01:33:10.160 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.094+0000 7fd58121c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:33:10.160 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.158+0000 7fd58121c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:33:10.286 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.010+0000 7f0df5a95040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:33:10.286 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.071+0000 7f0df5a95040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:33:10.287 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.139+0000 7f0df5a95040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:33:10.287 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.284+0000 7f0df5a95040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:33:10.303 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.026+0000 7fb2f7816040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T01:33:10.303 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.087+0000 7fb2f7816040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T01:33:10.303 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.154+0000 7fb2f7816040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:33:10.303 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.300+0000 7fb2f7816040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:33:10.445 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.228+0000 7fd58121c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T01:33:10.445 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.382+0000 7fd58121c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T01:33:10.445 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.444+0000 7fd58121c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi064.front.sepia.ceph.com 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-19T01:33:10.577 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-19T01:33:10.578 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-19T01:33:10.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:10.578 DEBUG:teuthology.orchestra.run.smithi064:> ip addr 2024-08-19T01:33:10.598 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.346+0000 7f0df5a95040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:33:10.598 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.412+0000 7f0df5a95040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:33:10.598 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.596+0000 7f0df5a95040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:33:10.612 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.361+0000 7fb2f7816040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T01:33:10.612 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.427+0000 7fb2f7816040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:33:10.612 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.610+0000 7fb2f7816040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout: inet 127.0.0.1/8 scope host lo 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 ::1/128 scope host 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-19T01:33:10.634 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:78 brd ff:ff:ff:ff:ff:ff 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f0 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:79 brd ff:ff:ff:ff:ff:ff 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f1 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:de brd ff:ff:ff:ff:ff:ff 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:df brd ff:ff:ff:ff:ff:ff 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: inet 172.21.15.64/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft 42286sec preferred_lft 42286sec 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 fe80::ec4:7aff:feb6:f8df/64 scope link noprefixroute 2024-08-19T01:33:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-08-19T01:33:10.636 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-08-19T01:33:10.636 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:33:10.636 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:33:10.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add name ceph-brx type bridge 2024-08-19T01:33:10.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr flush dev ceph-brx 2024-08-19T01:33:10.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set ceph-brx up 2024-08-19T01:33:10.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-08-19T01:33:10.636 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:33:10.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1413010870' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-08-19T01:33:10.711 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.512+0000 7fd58121c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T01:33:10.711 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.709+0000 7fd58121c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T01:33:10.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:33:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:10.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:10.776 DEBUG:teuthology.orchestra.run.smithi064:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-19T01:33:10.841 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-08-19T01:33:10.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:10.843 DEBUG:teuthology.orchestra.run.smithi064:> route 2024-08-19T01:33:10.899 INFO:teuthology.orchestra.run.smithi064.stdout:Kernel IP routing table 2024-08-19T01:33:10.900 INFO:teuthology.orchestra.run.smithi064.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-19T01:33:10.900 INFO:teuthology.orchestra.run.smithi064.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-08-19T01:33:10.900 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-08-19T01:33:10.900 INFO:teuthology.orchestra.run.smithi064.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-19T01:33:10.900 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:33:10.900 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:33:10.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-08-19T01:33:10.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-08-19T01:33:10.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-08-19T01:33:10.900 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:33:10.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1413010870' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-08-19T01:33:10.915 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.658+0000 7f0df5a95040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:33:10.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.719+0000 7f0df5a95040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:33:10.916 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.788+0000 7f0df5a95040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:33:10.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1413010870' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-08-19T01:33:10.949 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.672+0000 7fb2f7816040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:33:10.949 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.732+0000 7fb2f7816040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:33:10.949 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:10.800+0000 7fb2f7816040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:33:10.984 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.780+0000 7fd58121c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T01:33:10.985 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.844+0000 7fd58121c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T01:33:10.985 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:10.915+0000 7fd58121c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T01:33:10.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:33:11.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:11.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:11.029 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-08-19T01:33:11.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:11.045 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list-id 2024-08-19T01:33:11.100 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:33:11.100 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:33:11.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-19T01:33:11.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-08-19T01:33:11.100 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:33:11.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:33:11.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:11.219 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-08-19T01:33:11.219 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:33:11.219 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:33:11.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-08-19T01:33:11.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-08-19T01:33:11.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-08-19T01:33:11.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-08-19T01:33:11.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-08-19T01:33:11.220 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:33:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:33:11.277 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:11.140+0000 7fd58121c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:33:11.277 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:11.204+0000 7fd58121c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:33:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:11.338 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-19T01:33:11.338 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-08-19T01:33:11.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set brx.0 up 2024-08-19T01:33:11.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set dev brx.0 master ceph-brx 2024-08-19T01:33:11.338 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-08-19T01:33:11.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:33:11.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:11.415 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-19T01:33:11.415 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:11.416 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:10.999+0000 7f0df5a95040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:33:11.416 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:11.058+0000 7f0df5a95040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:33:11.448 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:11.012+0000 7fb2f7816040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T01:33:11.449 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:11.071+0000 7fb2f7816040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T01:33:11.450 INFO:teuthology.orchestra.run.smithi064.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-19T01:33:11.450 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-19T01:33:11.450 DEBUG:teuthology.orchestra.run.smithi064:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:11.505 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-08-19T01:33:11.505 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-08-19T01:33:11.505 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/9de8b1b3-1cf7-4bdf-a4f3-7b1fad68ee8a 2024-08-19T01:33:11.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:11.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/9de8b1b3-1cf7-4bdf-a4f3-7b1fad68ee8a /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-08-19T01:33:11.588 INFO:teuthology.orchestra.run.smithi064.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-08-19T01:33:11.589 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-08-19T01:33:11.713 INFO:teuthology.orchestra.run.smithi064.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-08-19T01:33:11.713 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-08-19T01:33:11.714 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: resolved to: "172.21.15.64:3300,172.21.15.123:3300,172.21.15.161:3300" 2024-08-19T01:33:11.714 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: trying mount with new device syntax: 0@88c1a048-5dc9-11ef-bccf-c7b262605968.cephfs=/volumes/qa/sv_1/9de8b1b3-1cf7-4bdf-a4f3-7b1fad68ee8a 2024-08-19T01:33:11.714 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.64:3300/172.21.15.123:3300/172.21.15.161:3300" will pass to kernel 2024-08-19T01:33:11.715 INFO:teuthology.orchestra.run.smithi064.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-19T01:33:11.715 INFO:teuthology.orchestra.run.smithi064.stdout: You just mounted a file system that supports labels which does not 2024-08-19T01:33:11.715 INFO:teuthology.orchestra.run.smithi064.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-19T01:33:11.715 INFO:teuthology.orchestra.run.smithi064.stdout: applications will generate AVC messages and not be allowed access to 2024-08-19T01:33:11.715 INFO:teuthology.orchestra.run.smithi064.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-19T01:33:11.716 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-19T01:33:11.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:33:11.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:11.790 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-19T01:33:11.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-19T01:33:11.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:11.886 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:11.571+0000 7f0df5a95040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:33:11.888 INFO:teuthology.orchestra.run.smithi064.stdout:client15267 2024-08-19T01:33:11.888 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:33:11.888 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-19T01:33:11.901 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:11.586+0000 7fb2f7816040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:33:11.906 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-19T01:33:11.984 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:11.764+0000 7fd58121c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T01:33:11.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-19T01:33:11.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:12.004 INFO:teuthology.orchestra.run.smithi064.stdout:client15267 2024-08-19T01:33:12.004 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-19T01:33:12.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-19T01:33:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:12.101 INFO:teuthology.orchestra.run.smithi064.stdout:client15267 2024-08-19T01:33:12.101 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968.client15267/status') 2024-08-19T01:33:12.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-19T01:33:12.160 INFO:teuthology.orchestra.run.smithi064.stdout:instance: client.15267 (3)192.168.0.1:0/2183969783 2024-08-19T01:33:12.161 INFO:teuthology.orchestra.run.smithi064.stdout:blocklisted: false 2024-08-19T01:33:12.161 INFO:teuthology.orchestra.run.smithi064.stderr:0+1 records in 2024-08-19T01:33:12.161 INFO:teuthology.orchestra.run.smithi064.stderr:0+1 records out 2024-08-19T01:33:12.161 INFO:teuthology.orchestra.run.smithi064.stderr:70 bytes copied, 2.3446e-05 s, 3.0 MB/s 2024-08-19T01:33:12.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-19T01:33:12.165 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:11.885+0000 7f0df5a95040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:33:12.168 INFO:teuthology.run_tasks:Running task exec... 2024-08-19T01:33:12.178 INFO:teuthology.task.exec:Executing custom commands... 2024-08-19T01:33:12.178 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:33:12.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-08-19T01:33:12.198 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:11.898+0000 7fb2f7816040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:33:12.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.280+0000 7fb64410c640 1 -- 172.21.15.64:0/440899863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c101960 msgr2=0x7fb63c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.280+0000 7fb64410c640 1 --2- 172.21.15.64:0/440899863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c101960 0x7fb63c101de0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009900 tx=0x7fb62c02a410 comp rx=0 tx=0).stop 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- 172.21.15.64:0/440899863 shutdown_connections 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 --2- 172.21.15.64:0/440899863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb63c102320 0x7fb63c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 --2- 172.21.15.64:0/440899863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c101960 0x7fb63c101de0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 --2- 172.21.15.64:0/440899863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb63c100770 0x7fb63c100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- 172.21.15.64:0/440899863 >> 172.21.15.64:0/440899863 conn(0x7fb63c0fbf10 msgr2=0x7fb63c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- 172.21.15.64:0/440899863 shutdown_connections 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- 172.21.15.64:0/440899863 wait complete. 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 Processor -- start 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- start start 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c100770 0x7fb63c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb63c101960 0x7fb63c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb63c102320 0x7fb63c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c10d540 con 0x7fb63c100770 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c10d3c0 con 0x7fb63c102320 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb64410c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c10d6c0 con 0x7fb63c101960 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.281+0000 7fb641ea8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c100770 0x7fb63c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c100770 0x7fb63c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35246/0 (socket says 172.21.15.64:35246) 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 -- 172.21.15.64:0/509840783 learned_addr learned my addr 172.21.15.64:0/509840783 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb6416a7640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb63c101960 0x7fb63c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb6426a9640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb63c102320 0x7fb63c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 -- 172.21.15.64:0/509840783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb63c101960 msgr2=0x7fb63c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb63c101960 0x7fb63c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 -- 172.21.15.64:0/509840783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb63c102320 msgr2=0x7fb63c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb63c102320 0x7fb63c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 -- 172.21.15.64:0/509840783 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63c19f700 con 0x7fb63c100770 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb641ea8640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c100770 0x7fb63c197ae0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb63800e9c0 tx=0x7fb63800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:12.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb632ffd640 1 -- 172.21.15.64:0/509840783 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63800cd80 con 0x7fb63c100770 2024-08-19T01:33:12.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb632ffd640 1 -- 172.21.15.64:0/509840783 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6380035f0 con 0x7fb63c100770 2024-08-19T01:33:12.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb63c19f9f0 con 0x7fb63c100770 2024-08-19T01:33:12.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb632ffd640 1 -- 172.21.15.64:0/509840783 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638010690 con 0x7fb63c100770 2024-08-19T01:33:12.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.282+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb63c19ff60 con 0x7fb63c100770 2024-08-19T01:33:12.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.283+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb63c100b70 con 0x7fb63c100770 2024-08-19T01:33:12.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.285+0000 7fb632ffd640 1 -- 172.21.15.64:0/509840783 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fb6380037a0 con 0x7fb63c100770 2024-08-19T01:33:12.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.285+0000 7fb632ffd640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 0x7fb6280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.285+0000 7fb6416a7640 1 -- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 msgr2=0x7fb6280a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2663741075 2024-08-19T01:33:12.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.285+0000 7fb632ffd640 1 -- 172.21.15.64:0/509840783 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb6380cf7d0 con 0x7fb63c100770 2024-08-19T01:33:12.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.285+0000 7fb6416a7640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 0x7fb6280a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:33:12.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.289+0000 7fb632ffd640 1 -- 172.21.15.64:0/509840783 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb63809c360 con 0x7fb63c100770 2024-08-19T01:33:12.484 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:12.096+0000 7fd58121c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T01:33:12.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.485+0000 7fb6416a7640 1 -- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 msgr2=0x7fb6280a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2663741075 2024-08-19T01:33:12.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.485+0000 7fb6416a7640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 0x7fb6280a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-19T01:33:12.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.502+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fb63c107a90 con 0x7fb63c100770 2024-08-19T01:33:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.554+0000 7fb632ffd640 1 -- 172.21.15.64:0/509840783 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7fb63809bc60 con 0x7fb63c100770 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 msgr2=0x7fb6280a51c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 0x7fb6280a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c100770 msgr2=0x7fb63c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c100770 0x7fb63c197ae0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb63800e9c0 tx=0x7fb63800ee80 comp rx=0 tx=0).stop 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 shutdown_connections 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7fb6280a2d00 0x7fb6280a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb63c102320 0x7fb63c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb63c101960 0x7fb63c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 --2- 172.21.15.64:0/509840783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb63c100770 0x7fb63c197ae0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.557+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 >> 172.21.15.64:0/509840783 conn(0x7fb63c0fbf10 msgr2=0x7fb63c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.558+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 shutdown_connections 2024-08-19T01:33:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.558+0000 7fb64410c640 1 -- 172.21.15.64:0/509840783 wait complete. 2024-08-19T01:33:12.562 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:12.300+0000 7f0df5a95040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:33:12.562 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:12.355+0000 7f0df5a95040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:33:12.562 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:33:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y[23199]: 2024-08-19T01:33:12.414+0000 7f0df5a95040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:33:12.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 -- 172.21.15.64:0/1710924932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294101960 msgr2=0x7f7294101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 --2- 172.21.15.64:0/1710924932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294101960 0x7f7294101de0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7284009900 tx=0x7f728402a410 comp rx=0 tx=0).stop 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 -- 172.21.15.64:0/1710924932 shutdown_connections 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 --2- 172.21.15.64:0/1710924932 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7294102320 0x7f7294108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 --2- 172.21.15.64:0/1710924932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294101960 0x7f7294101de0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 --2- 172.21.15.64:0/1710924932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7294100770 0x7f7294100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 -- 172.21.15.64:0/1710924932 >> 172.21.15.64:0/1710924932 conn(0x7f72940fbf10 msgr2=0x7f72940fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 -- 172.21.15.64:0/1710924932 shutdown_connections 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.678+0000 7f729babc640 1 -- 172.21.15.64:0/1710924932 wait complete. 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 Processor -- start 2024-08-19T01:33:12.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 -- start start 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7294100770 0x7f729406a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7294101960 0x7f729419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294102320 0x7f729419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f729410c3d0 con 0x7f7294102320 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f729410c250 con 0x7f7294101960 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729babc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f729410c550 con 0x7f7294100770 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729a059640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294102320 0x7f729419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729a059640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294102320 0x7f729419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35266/0 (socket says 172.21.15.64:35266) 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f729a059640 1 -- 172.21.15.64:0/2113477547 learned_addr learned my addr 172.21.15.64:0/2113477547 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f7299858640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7294100770 0x7f729406a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.679+0000 7f7299057640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7294101960 0x7f729419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729a059640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7294100770 msgr2=0x7f729406a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729a059640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7294100770 0x7f729406a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729a059640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7294101960 msgr2=0x7f729419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729a059640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7294101960 0x7f729419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729a059640 1 -- 172.21.15.64:0/2113477547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729419f710 con 0x7f7294102320 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729a059640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294102320 0x7f729419ee50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f728c00eaa0 tx=0x7f728c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c00cdb0 con 0x7f7294102320 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f728c005020 con 0x7f7294102320 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729419fa00 con 0x7f7294102320 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c0116b0 con 0x7f7294102320 2024-08-19T01:33:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.680+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729419ff20 con 0x7f7294102320 2024-08-19T01:33:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.681+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7294107b60 con 0x7f7294102320 2024-08-19T01:33:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.682+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f728c005560 con 0x7f7294102320 2024-08-19T01:33:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.683+0000 7f728affd640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 0x7f72800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.683+0000 7f7299858640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 msgr2=0x7f72800a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2663741075 2024-08-19T01:33:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.683+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f728c0cf5e0 con 0x7f7294102320 2024-08-19T01:33:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.683+0000 7f7299858640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 0x7f72800a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-19T01:33:12.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.687+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f728c016030 con 0x7f7294102320 2024-08-19T01:33:12.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:12 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/509840783' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-08-19T01:33:12.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:12.314+0000 7fb2f7816040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:33:12.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:12.371+0000 7fb2f7816040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:33:12.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:33:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z[23420]: 2024-08-19T01:33:12.431+0000 7fb2f7816040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:33:12.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.884+0000 7f7299858640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 msgr2=0x7f72800a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2663741075 2024-08-19T01:33:12.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.884+0000 7f7299858640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 0x7f72800a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-19T01:33:12.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:12.889+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 --> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f7294062360 con 0x7f72800a2d00 2024-08-19T01:33:12.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:12 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/509840783' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-08-19T01:33:12.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:12 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/509840783' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-08-19T01:33:12.985 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:12.535+0000 7fd58121c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T01:33:12.985 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:12.598+0000 7fd58121c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T01:33:12.985 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:12.661+0000 7fd58121c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T01:33:13.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:13.285+0000 7f7299858640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 msgr2=0x7f72800a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2663741075 2024-08-19T01:33:13.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:13.285+0000 7f7299858640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 0x7f72800a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-19T01:33:13.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:13.574+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 20) ==== 137850+0+0 (secure 0 0 0) 0x7f728c098190 con 0x7f7294102320 2024-08-19T01:33:13.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:13.825+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7f728c0ccd60 con 0x7f7294102320 2024-08-19T01:33:13.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:13.825+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 msgr2=0x7f72800a51c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T01:33:13.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:13.825+0000 7f728affd640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/2663741075,v1:172.21.15.64:6801/2663741075] conn(0x7f72800a2d00 0x7f72800a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:13.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/509840783' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: Standby manager daemon y restarted 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: Standby manager daemon y started 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: Standby manager daemon z restarted 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: Standby manager daemon z started 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-08-19T01:33:13.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:13 smithi123 ceph-mon[21829]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:33:13.947 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/509840783' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: Standby manager daemon y restarted 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: Standby manager daemon y started 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: Standby manager daemon z restarted 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: Standby manager daemon z started 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:33:13.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:13 smithi064 ceph-mon[18515]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/509840783' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: Standby manager daemon y restarted 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: Standby manager daemon y started 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-19T01:33:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.123:0/2636569000' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:33:13.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: Standby manager daemon z restarted 2024-08-19T01:33:13.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: Standby manager daemon z started 2024-08-19T01:33:13.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-08-19T01:33:13.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T01:33:13.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-08-19T01:33:13.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:13 smithi161 ceph-mon[21757]: from='mgr.? 172.21.15.161:0/1448983720' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.083+0000 7fd4e7c60640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.083+0000 7fd4e7c60640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.083+0000 7fd4e7c60640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.083+0000 7fd4e7c60640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.083+0000 7fd4e7c60640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.085+0000 7fd4eac66640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.085+0000 7fd4eac66640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.085+0000 7fd4eac66640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.085+0000 7fd4eac66640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.085+0000 7fd4eac66640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:14.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-08-19T01:33:14.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: Active manager daemon x restarted 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: Activating manager daemon x 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: osdmap e94: 12 total, 12 up, 12 in 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: mgrmap e21: x(active, starting, since 0.0116596s), standbys: z, y 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-08-19T01:33:14.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:33:14.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: Manager daemon x is now available 2024-08-19T01:33:14.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:33:14.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:33:14.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:33:14.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:33:14.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:33:14.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.829+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mon.0 v2:172.21.15.64:3300/0 9 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f728c04deb0 con 0x7f7294102320 2024-08-19T01:33:14.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.830+0000 7f728affd640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f72800356a0 0x7f7280035a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:14.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.830+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f728c09a810 con 0x7f72800356a0 2024-08-19T01:33:14.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.834+0000 7f7299858640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f72800356a0 0x7f7280035a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:14.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.837+0000 7f7299858640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f72800356a0 0x7f7280035a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f72941017c0 tx=0x7f729000b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:14.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.851+0000 7f728affd640 1 -- 172.21.15.64:0/2113477547 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f728c09a810 con 0x7f72800356a0 2024-08-19T01:33:14.853 INFO:teuthology.orchestra.run.smithi064.stdout:{} 2024-08-19T01:33:14.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f72800356a0 msgr2=0x7f7280035a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f72800356a0 0x7f7280035a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f72941017c0 tx=0x7f729000b390 comp rx=0 tx=0).stop 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294102320 msgr2=0x7f729419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 --2- 172.21.15.64:0/2113477547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7294102320 0x7f729419ee50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f728c00eaa0 tx=0x7f728c00ef60 comp rx=0 tx=0).stop 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729a059640 1 -- 172.21.15.64:0/2113477547 reap_dead start 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 shutdown_connections 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 >> 172.21.15.64:0/2113477547 conn(0x7f72940fbf10 msgr2=0x7f7294104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 shutdown_connections 2024-08-19T01:33:14.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:14.853+0000 7f729babc640 1 -- 172.21.15.64:0/2113477547 wait complete. 2024-08-19T01:33:14.867 INFO:teuthology.run_tasks:Running task check-counter... 2024-08-19T01:33:14.883 INFO:teuthology.run_tasks:Running task exec... 2024-08-19T01:33:14.914 INFO:teuthology.task.exec:Executing custom commands... 2024-08-19T01:33:14.914 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:33:14.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: Active manager daemon x restarted 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: Activating manager daemon x 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: osdmap e94: 12 total, 12 up, 12 in 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: mgrmap e21: x(active, starting, since 0.0116596s), standbys: z, y 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-08-19T01:33:14.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:33:14.917 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: Manager daemon x is now available 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:33:14.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:33:14.919 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:33:14.919 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:33:14.919 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:14 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:33:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-08-19T01:33:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: Active manager daemon x restarted 2024-08-19T01:33:14.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: Activating manager daemon x 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: osdmap e94: 12 total, 12 up, 12 in 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: mgrmap e21: x(active, starting, since 0.0116596s), standbys: z, y 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-08-19T01:33:14.950 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T01:33:14.951 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: Manager daemon x is now available 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:33:14.952 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:14 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-19T01:33:15.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 -- 172.21.15.64:0/772032354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 msgr2=0x7f54b8107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/772032354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b8107010 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009900 tx=0x7f54a802a410 comp rx=0 tx=0).stop 2024-08-19T01:33:15.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 -- 172.21.15.64:0/772032354 shutdown_connections 2024-08-19T01:33:15.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/772032354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b8107550 0x7f54b81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/772032354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b8107010 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/772032354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b81022f0 0x7f54b81046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 -- 172.21.15.64:0/772032354 >> 172.21.15.64:0/772032354 conn(0x7f54b80fbf10 msgr2=0x7f54b80fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:15.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 -- 172.21.15.64:0/772032354 shutdown_connections 2024-08-19T01:33:15.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.019+0000 7f54bdbdf640 1 -- 172.21.15.64:0/772032354 wait complete. 2024-08-19T01:33:15.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 Processor -- start 2024-08-19T01:33:15.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 -- start start 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b81022f0 0x7f54b8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b8107550 0x7f54b819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f54b810d500 con 0x7f54b8104c20 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f54b810d380 con 0x7f54b8107550 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54bdbdf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f54b810d680 con 0x7f54b81022f0 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54b6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54b6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b81980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35398/0 (socket says 172.21.15.64:35398) 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54b6ffd640 1 -- 172.21.15.64:0/2550171806 learned_addr learned my addr 172.21.15.64:0/2550171806 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.020+0000 7f54b77fe640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b81022f0 0x7f54b8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b7fff640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b8107550 0x7f54b819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b6ffd640 1 -- 172.21.15.64:0/2550171806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b81022f0 msgr2=0x7f54b8197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b6ffd640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b81022f0 0x7f54b8197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b6ffd640 1 -- 172.21.15.64:0/2550171806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b8107550 msgr2=0x7f54b819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b6ffd640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b8107550 0x7f54b819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b6ffd640 1 -- 172.21.15.64:0/2550171806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b819f7b0 con 0x7f54b8104c20 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b6ffd640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b81980d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f54a8031610 tx=0x7f54a80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b4ff9640 1 -- 172.21.15.64:0/2550171806 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a803f030 con 0x7f54b8104c20 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b4ff9640 1 -- 172.21.15.64:0/2550171806 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a8003ce0 con 0x7f54b8104c20 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b819fa40 con 0x7f54b8104c20 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54b4ff9640 1 -- 172.21.15.64:0/2550171806 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8035070 con 0x7f54b8104c20 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.021+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b819ff30 con 0x7f54b8104c20 2024-08-19T01:33:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.022+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b80fda00 con 0x7f54b8104c20 2024-08-19T01:33:15.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.023+0000 7f54b4ff9640 1 -- 172.21.15.64:0/2550171806 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f54a8042030 con 0x7f54b8104c20 2024-08-19T01:33:15.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.024+0000 7f54b4ff9640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f54a40a2a40 0x7f54a40a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.024+0000 7f54b77fe640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f54a40a2a40 0x7f54a40a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.024+0000 7f54b4ff9640 1 -- 172.21.15.64:0/2550171806 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f54a80ef840 con 0x7f54b8104c20 2024-08-19T01:33:15.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.025+0000 7f54b77fe640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f54a40a2a40 0x7f54a40a4f00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f54a0005d00 tx=0x7f54a000b410 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:15.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.031+0000 7f54b4ff9640 1 -- 172.21.15.64:0/2550171806 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f54a80bbf70 con 0x7f54b8104c20 2024-08-19T01:33:15.204 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.842+0000 7fd4e5c1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:15.204 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.842+0000 7fd4e5c1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:15.204 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.842+0000 7fd4e5c1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:15.204 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.842+0000 7fd4e5c1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:15.204 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:14.842+0000 7fd4e5c1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:33:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.213+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f54b8062b90 con 0x7f54b8104c20 2024-08-19T01:33:15.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.217+0000 7f54b4ff9640 1 -- 172.21.15.64:0/2550171806 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v28)=0 v28) ==== 129+0+0 (secure 0 0 0) 0x7f54a80c0270 con 0x7f54b8104c20 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f54a40a2a40 msgr2=0x7f54a40a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f54a40a2a40 0x7f54a40a4f00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f54a0005d00 tx=0x7f54a000b410 comp rx=0 tx=0).stop 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 msgr2=0x7f54b81980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b81980d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f54a8031610 tx=0x7f54a80043b0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 shutdown_connections 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f54a40a2a40 0x7f54a40a4f00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b8107550 0x7f54b819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54b8104c20 0x7f54b81980d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 --2- 172.21.15.64:0/2550171806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b81022f0 0x7f54b8197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 >> 172.21.15.64:0/2550171806 conn(0x7f54b80fbf10 msgr2=0x7f54b8102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:15.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 shutdown_connections 2024-08-19T01:33:15.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.219+0000 7f54bdbdf640 1 -- 172.21.15.64:0/2550171806 wait complete. 2024-08-19T01:33:15.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-08-19T01:33:15.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 -- 172.21.15.64:0/2054268406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 msgr2=0x7fe100101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 --2- 172.21.15.64:0/2054268406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100101de0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009900 tx=0x7fe0f402a410 comp rx=0 tx=0).stop 2024-08-19T01:33:15.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 -- 172.21.15.64:0/2054268406 shutdown_connections 2024-08-19T01:33:15.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 --2- 172.21.15.64:0/2054268406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe100102320 0x7fe100108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 --2- 172.21.15.64:0/2054268406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100101de0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 --2- 172.21.15.64:0/2054268406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe100100770 0x7fe100100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 -- 172.21.15.64:0/2054268406 >> 172.21.15.64:0/2054268406 conn(0x7fe1000fbf10 msgr2=0x7fe1000fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:15.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 -- 172.21.15.64:0/2054268406 shutdown_connections 2024-08-19T01:33:15.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.333+0000 7fe10672e640 1 -- 172.21.15.64:0/2054268406 wait complete. 2024-08-19T01:33:15.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 Processor -- start 2024-08-19T01:33:15.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 -- start start 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe100100770 0x7fe1001969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe100102320 0x7fe10019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe10010c3d0 con 0x7fe100101960 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe10010c250 con 0x7fe100100770 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.334+0000 7fe10672e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe10010c550 con 0x7fe100102320 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100196f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35404/0 (socket says 172.21.15.64:35404) 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 -- 172.21.15.64:0/1076898787 learned_addr learned my addr 172.21.15.64:0/1076898787 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0fffff640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe100100770 0x7fe1001969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe104ccb640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe100102320 0x7fe10019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 -- 172.21.15.64:0/1076898787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe100102320 msgr2=0x7fe10019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe100102320 0x7fe10019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 -- 172.21.15.64:0/1076898787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe100100770 msgr2=0x7fe1001969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe100100770 0x7fe1001969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 -- 172.21.15.64:0/1076898787 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10019e5f0 con 0x7fe100101960 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0ff7fe640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100196f10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009230 tx=0x7fe0f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0fd7fa640 1 -- 172.21.15.64:0/1076898787 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f403f030 con 0x7fe100101960 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0fd7fa640 1 -- 172.21.15.64:0/1076898787 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0f4003ce0 con 0x7fe100101960 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe0fd7fa640 1 -- 172.21.15.64:0/1076898787 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4035070 con 0x7fe100101960 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10019e880 con 0x7fe100101960 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.335+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe10019ed70 con 0x7fe100101960 2024-08-19T01:33:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.337+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe100107a90 con 0x7fe100101960 2024-08-19T01:33:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.337+0000 7fe0fd7fa640 1 -- 172.21.15.64:0/1076898787 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fe0f4042030 con 0x7fe100101960 2024-08-19T01:33:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.338+0000 7fe0fd7fa640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe0e80a2a40 0x7fe0e80a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.338+0000 7fe0fffff640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe0e80a2a40 0x7fe0e80a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.338+0000 7fe0fd7fa640 1 -- 172.21.15.64:0/1076898787 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe0f40ef690 con 0x7fe100101960 2024-08-19T01:33:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.339+0000 7fe0fffff640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe0e80a2a40 0x7fe0e80a4f00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1001017c0 tx=0x7fe0f0005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.342+0000 7fe0fd7fa640 1 -- 172.21.15.64:0/1076898787 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe0f40bc410 con 0x7fe100101960 2024-08-19T01:33:15.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.536+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fe100062b20 con 0x7fe100101960 2024-08-19T01:33:15.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.540+0000 7fe0fd7fa640 1 -- 172.21.15.64:0/1076898787 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7fe0f40bbd10 con 0x7fe100101960 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe0e80a2a40 msgr2=0x7fe0e80a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe0e80a2a40 0x7fe0e80a4f00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1001017c0 tx=0x7fe0f0005f90 comp rx=0 tx=0).stop 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 msgr2=0x7fe100196f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100196f10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009230 tx=0x7fe0f40042f0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 shutdown_connections 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe0e80a2a40 0x7fe0e80a4f00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe100102320 0x7fe10019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe100101960 0x7fe100196f10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 --2- 172.21.15.64:0/1076898787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe100100770 0x7fe1001969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 >> 172.21.15.64:0/1076898787 conn(0x7fe1000fbf10 msgr2=0x7fe100104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 shutdown_connections 2024-08-19T01:33:15.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.542+0000 7fe10672e640 1 -- 172.21.15.64:0/1076898787 wait complete. 2024-08-19T01:33:15.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-08-19T01:33:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 -- 172.21.15.64:0/266704781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec0ffeb0 msgr2=0x7f76ec1002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 --2- 172.21.15.64:0/266704781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec0ffeb0 0x7f76ec1002b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f76dc009900 tx=0x7f76dc02a410 comp rx=0 tx=0).stop 2024-08-19T01:33:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 -- 172.21.15.64:0/266704781 shutdown_connections 2024-08-19T01:33:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 --2- 172.21.15.64:0/266704781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76ec101a50 0x7f76ec108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 --2- 172.21.15.64:0/266704781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76ec101090 0x7f76ec101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 --2- 172.21.15.64:0/266704781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec0ffeb0 0x7f76ec1002b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 -- 172.21.15.64:0/266704781 >> 172.21.15.64:0/266704781 conn(0x7f76ec0fb660 msgr2=0x7f76ec0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:15.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 -- 172.21.15.64:0/266704781 shutdown_connections 2024-08-19T01:33:15.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.673+0000 7f76f191a640 1 -- 172.21.15.64:0/266704781 wait complete. 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.674+0000 7f76f191a640 1 Processor -- start 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.674+0000 7f76f191a640 1 -- start start 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.674+0000 7f76f191a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76ec0ffeb0 0x7f76ec199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.674+0000 7f76f191a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec101090 0x7f76ec19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.674+0000 7f76f191a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76ec101a50 0x7f76ec1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.674+0000 7f76f191a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec10c400 con 0x7f76ec101090 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76f191a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec10c280 con 0x7f76ec0ffeb0 2024-08-19T01:33:15.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76f191a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec10c580 con 0x7f76ec101a50 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec101090 0x7f76ec19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec101090 0x7f76ec19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35428/0 (socket says 172.21.15.64:35428) 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 -- 172.21.15.64:0/2481614044 learned_addr learned my addr 172.21.15.64:0/2481614044 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76eaffd640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76ec0ffeb0 0x7f76ec199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76eb7fe640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76ec101a50 0x7f76ec1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 -- 172.21.15.64:0/2481614044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76ec101a50 msgr2=0x7f76ec1a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76ec101a50 0x7f76ec1a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 -- 172.21.15.64:0/2481614044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76ec0ffeb0 msgr2=0x7f76ec199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76ec0ffeb0 0x7f76ec199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 -- 172.21.15.64:0/2481614044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ec1a1940 con 0x7f76ec101090 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76eaffd640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76ec0ffeb0 0x7f76ec199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:33:15.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76ea7fc640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec101090 0x7f76ec19a270 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f76e000d950 tx=0x7f76e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76f0918640 1 -- 172.21.15.64:0/2481614044 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0017030 con 0x7f76ec101090 2024-08-19T01:33:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76f0918640 1 -- 172.21.15.64:0/2481614044 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e0004020 con 0x7f76ec101090 2024-08-19T01:33:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76ec1a1c30 con 0x7f76ec101090 2024-08-19T01:33:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76f0918640 1 -- 172.21.15.64:0/2481614044 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e00047a0 con 0x7f76ec101090 2024-08-19T01:33:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.675+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ec1a2150 con 0x7f76ec101090 2024-08-19T01:33:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.677+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76ec1a2320 con 0x7f76ec101090 2024-08-19T01:33:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.678+0000 7f76f0918640 1 -- 172.21.15.64:0/2481614044 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f76e00041d0 con 0x7f76ec101090 2024-08-19T01:33:15.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.678+0000 7f76f0918640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f76d40a2ac0 0x7f76d40a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:15.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.678+0000 7f76eaffd640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f76d40a2ac0 0x7f76d40a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:15.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.678+0000 7f76f0918640 1 -- 172.21.15.64:0/2481614044 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f76e00ceba0 con 0x7f76ec101090 2024-08-19T01:33:15.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.679+0000 7f76eaffd640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f76d40a2ac0 0x7f76d40a4f80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f76dc005ab0 tx=0x7f76dc000bc0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:15.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.682+0000 7f76f0918640 1 -- 172.21.15.64:0/2481614044 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f76e009b8e0 con 0x7f76ec101090 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:15.575+0000 7fd4d03f2640 -1 log_channel(cephadm) log [ERR] : [19/Aug/2024:01:33:15] ENGINE Error in HTTPServer.serve 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: Traceback (most recent call last): 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._connections.run(self.expiration_interval) 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._run(expiration_interval) 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:33:15.830 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: s = self.context.wrap_socket( 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: return self.sslsocket_class._create( 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self.do_handshake() 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: self._sslobj.do_handshake() 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:33:15.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:33:15.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.878+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f76ec1a24f0 con 0x7f76ec101090 2024-08-19T01:33:15.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.883+0000 7f76f0918640 1 -- 172.21.15.64:0/2481614044 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f76e00b9e20 con 0x7f76ec101090 2024-08-19T01:33:15.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f76d40a2ac0 msgr2=0x7f76d40a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f76d40a2ac0 0x7f76d40a4f80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f76dc005ab0 tx=0x7f76dc000bc0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec101090 msgr2=0x7f76ec19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec101090 0x7f76ec19a270 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f76e000d950 tx=0x7f76e000de10 comp rx=0 tx=0).stop 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 shutdown_connections 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f76d40a2ac0 0x7f76d40a4f80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76ec101a50 0x7f76ec1a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f76ec101090 0x7f76ec19a270 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 --2- 172.21.15.64:0/2481614044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76ec0ffeb0 0x7f76ec199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 >> 172.21.15.64:0/2481614044 conn(0x7f76ec0fb660 msgr2=0x7f76ec0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 shutdown_connections 2024-08-19T01:33:15.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:15.886+0000 7f76f191a640 1 -- 172.21.15.64:0/2481614044 wait complete. 2024-08-19T01:33:15.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.016+0000 7fb8d7634640 1 -- 172.21.15.64:0/1897797085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 msgr2=0x7fb8d0107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.016+0000 7fb8d7634640 1 --2- 172.21.15.64:0/1897797085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 0x7fb8d0107020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009860 tx=0x7fb8c002a340 comp rx=0 tx=0).stop 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.016+0000 7fb8d7634640 1 -- 172.21.15.64:0/1897797085 shutdown_connections 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.016+0000 7fb8d7634640 1 --2- 172.21.15.64:0/1897797085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0107560 0x7fb8d01099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.016+0000 7fb8d7634640 1 --2- 172.21.15.64:0/1897797085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 0x7fb8d0107020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.016+0000 7fb8d7634640 1 --2- 172.21.15.64:0/1897797085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8d0102300 0x7fb8d01046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.016+0000 7fb8d7634640 1 -- 172.21.15.64:0/1897797085 >> 172.21.15.64:0/1897797085 conn(0x7fb8d00fbee0 msgr2=0x7fb8d00fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 -- 172.21.15.64:0/1897797085 shutdown_connections 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 -- 172.21.15.64:0/1897797085 wait complete. 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 Processor -- start 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 -- start start 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0102300 0x7fb8d0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 0x7fb8d01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8d0107560 0x7fb8d019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d010d530 con 0x7fb8d0102300 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d010d3b0 con 0x7fb8d0107560 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d7634640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d010d6b0 con 0x7fb8d0104c30 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d53d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0102300 0x7fb8d0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.017+0000 7fb8d53d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0102300 0x7fb8d0197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35472/0 (socket says 172.21.15.64:35472) 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d4bcf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 0x7fb8d01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d53d0640 1 -- 172.21.15.64:0/402617096 learned_addr learned my addr 172.21.15.64:0/402617096 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d5bd1640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8d0107560 0x7fb8d019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d4bcf640 1 -- 172.21.15.64:0/402617096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8d0107560 msgr2=0x7fb8d019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d4bcf640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8d0107560 0x7fb8d019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d4bcf640 1 -- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0102300 msgr2=0x7fb8d0197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d4bcf640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0102300 0x7fb8d0197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d4bcf640 1 -- 172.21.15.64:0/402617096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d019f7d0 con 0x7fb8d0104c30 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d53d0640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0102300 0x7fb8d0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d4bcf640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 0x7fb8d01980f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0035040 tx=0x7fb8c0033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8c67fc640 1 -- 172.21.15.64:0/402617096 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0037040 con 0x7fb8d0104c30 2024-08-19T01:33:16.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8c67fc640 1 -- 172.21.15.64:0/402617096 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8c0004040 con 0x7fb8d0104c30 2024-08-19T01:33:16.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8d019fa60 con 0x7fb8d0104c30 2024-08-19T01:33:16.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8c67fc640 1 -- 172.21.15.64:0/402617096 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00055c0 con 0x7fb8d0104c30 2024-08-19T01:33:16.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.018+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8d019ff50 con 0x7fb8d0104c30 2024-08-19T01:33:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.019+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8d00fd9d0 con 0x7fb8d0104c30 2024-08-19T01:33:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.020+0000 7fb8c67fc640 1 -- 172.21.15.64:0/402617096 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fb8c0042020 con 0x7fb8d0104c30 2024-08-19T01:33:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.021+0000 7fb8c67fc640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb8bc0a2a40 0x7fb8bc0a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.021+0000 7fb8c67fc640 1 -- 172.21.15.64:0/402617096 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb8c00eed30 con 0x7fb8d0104c30 2024-08-19T01:33:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.021+0000 7fb8d53d0640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb8bc0a2a40 0x7fb8bc0a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.022+0000 7fb8d53d0640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb8bc0a2a40 0x7fb8bc0a4f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc011500 tx=0x7fb8cc005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.024+0000 7fb8c67fc640 1 -- 172.21.15.64:0/402617096 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb8c0049020 con 0x7fb8d0104c30 2024-08-19T01:33:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: mgrmap e22: x(active, since 1.01781s), standbys: z, y 2024-08-19T01:33:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2550171806' entity='client.admin' 2024-08-19T01:33:16.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: [19/Aug/2024:01:33:15] ENGINE Bus STARTING 2024-08-19T01:33:16.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1076898787' entity='client.admin' 2024-08-19T01:33:16.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: [19/Aug/2024:01:33:15] ENGINE Serving on https://172.21.15.64:7150 2024-08-19T01:33:16.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: [19/Aug/2024:01:33:15] ENGINE Error in HTTPServer.serve 2024-08-19T01:33:16.166 INFO:journalctl@ceph.mon.b.smithi123.stdout: Traceback (most recent call last): 2024-08-19T01:33:16.166 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:33:16.166 INFO:journalctl@ceph.mon.b.smithi123.stdout: self._connections.run(self.expiration_interval) 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: self._run(expiration_interval) 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: s = self.context.wrap_socket( 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: return self.sslsocket_class._create( 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:33:16.167 INFO:journalctl@ceph.mon.b.smithi123.stdout: self.do_handshake() 2024-08-19T01:33:16.168 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:33:16.168 INFO:journalctl@ceph.mon.b.smithi123.stdout: self._sslobj.do_handshake() 2024-08-19T01:33:16.168 INFO:journalctl@ceph.mon.b.smithi123.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:33:16.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: [19/Aug/2024:01:33:15] ENGINE Serving on http://172.21.15.64:8765 2024-08-19T01:33:16.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:15 smithi123 ceph-mon[21829]: [19/Aug/2024:01:33:15] ENGINE Bus STARTED 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: mgrmap e22: x(active, since 1.01781s), standbys: z, y 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2550171806' entity='client.admin' 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: [19/Aug/2024:01:33:15] ENGINE Bus STARTING 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1076898787' entity='client.admin' 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: [19/Aug/2024:01:33:15] ENGINE Serving on https://172.21.15.64:7150 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: [19/Aug/2024:01:33:15] ENGINE Error in HTTPServer.serve 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout: Traceback (most recent call last): 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout: self._connections.run(self.expiration_interval) 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:33:16.199 INFO:journalctl@ceph.mon.c.smithi161.stdout: self._run(expiration_interval) 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: s = self.context.wrap_socket( 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: return self.sslsocket_class._create( 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: self.do_handshake() 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: self._sslobj.do_handshake() 2024-08-19T01:33:16.200 INFO:journalctl@ceph.mon.c.smithi161.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:33:16.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: [19/Aug/2024:01:33:15] ENGINE Serving on http://172.21.15.64:8765 2024-08-19T01:33:16.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:15 smithi161 ceph-mon[21757]: [19/Aug/2024:01:33:15] ENGINE Bus STARTED 2024-08-19T01:33:16.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.228+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb8d0062360 con 0x7fb8bc0a2a40 2024-08-19T01:33:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: mgrmap e22: x(active, since 1.01781s), standbys: z, y 2024-08-19T01:33:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2550171806' entity='client.admin' 2024-08-19T01:33:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: [19/Aug/2024:01:33:15] ENGINE Bus STARTING 2024-08-19T01:33:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1076898787' entity='client.admin' 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: [19/Aug/2024:01:33:15] ENGINE Serving on https://172.21.15.64:7150 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: [19/Aug/2024:01:33:15] ENGINE Error in HTTPServer.serve 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._connections.run(self.expiration_interval) 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._run(expiration_interval) 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T01:33:16.235 INFO:journalctl@ceph.mon.a.smithi064.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: s = self.context.wrap_socket( 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: return self.sslsocket_class._create( 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: self.do_handshake() 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._sslobj.do_handshake() 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T01:33:16.236 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: [19/Aug/2024:01:33:15] ENGINE Serving on http://172.21.15.64:8765 2024-08-19T01:33:16.237 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:15 smithi064 ceph-mon[18515]: [19/Aug/2024:01:33:15] ENGINE Bus STARTED 2024-08-19T01:33:16.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.349+0000 7fb8c67fc640 1 -- 172.21.15.64:0/402617096 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb8d0062360 con 0x7fb8bc0a2a40 2024-08-19T01:33:16.352 INFO:teuthology.orchestra.run.smithi064.stdout:Schedule set for path / 2024-08-19T01:33:16.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.352+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb8bc0a2a40 msgr2=0x7fb8bc0a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.352+0000 7fb8d7634640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb8bc0a2a40 0x7fb8bc0a4f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc011500 tx=0x7fb8cc005f90 comp rx=0 tx=0).stop 2024-08-19T01:33:16.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.352+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 msgr2=0x7fb8d01980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.352+0000 7fb8d7634640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 0x7fb8d01980f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0035040 tx=0x7fb8c0033d90 comp rx=0 tx=0).stop 2024-08-19T01:33:16.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.352+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 shutdown_connections 2024-08-19T01:33:16.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.352+0000 7fb8d7634640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb8bc0a2a40 0x7fb8bc0a4f00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.352+0000 7fb8d7634640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8d0107560 0x7fb8d019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.353+0000 7fb8d7634640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8d0104c30 0x7fb8d01980f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.353+0000 7fb8d7634640 1 --2- 172.21.15.64:0/402617096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8d0102300 0x7fb8d0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.353+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 >> 172.21.15.64:0/402617096 conn(0x7fb8d00fbee0 msgr2=0x7fb8d0102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:16.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.353+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 shutdown_connections 2024-08-19T01:33:16.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.353+0000 7fb8d7634640 1 -- 172.21.15.64:0/402617096 wait complete. 2024-08-19T01:33:16.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.472+0000 7fed041a8640 1 -- 172.21.15.64:0/1391107817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 msgr2=0x7fecfc100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.472+0000 7fed041a8640 1 --2- 172.21.15.64:0/1391107817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc100b70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009900 tx=0x7fecf802a410 comp rx=0 tx=0).stop 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.472+0000 7fed041a8640 1 -- 172.21.15.64:0/1391107817 shutdown_connections 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.472+0000 7fed041a8640 1 --2- 172.21.15.64:0/1391107817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecfc102320 0x7fecfc108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.472+0000 7fed041a8640 1 --2- 172.21.15.64:0/1391107817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecfc101960 0x7fecfc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.472+0000 7fed041a8640 1 --2- 172.21.15.64:0/1391107817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc100b70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 -- 172.21.15.64:0/1391107817 >> 172.21.15.64:0/1391107817 conn(0x7fecfc0fbf10 msgr2=0x7fecfc0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 -- 172.21.15.64:0/1391107817 shutdown_connections 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 -- 172.21.15.64:0/1391107817 wait complete. 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 Processor -- start 2024-08-19T01:33:16.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 -- start start 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecfc101960 0x7fecfc1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecfc102320 0x7fecfc19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc10d5a0 con 0x7fecfc100770 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc10d420 con 0x7fecfc101960 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed041a8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc10d720 con 0x7fecfc102320 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed01f44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.473+0000 7fed01f44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35498/0 (socket says 172.21.15.64:35498) 2024-08-19T01:33:16.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01f44640 1 -- 172.21.15.64:0/1586308354 learned_addr learned my addr 172.21.15.64:0/1586308354 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01743640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecfc101960 0x7fecfc1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed02745640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecfc102320 0x7fecfc19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01f44640 1 -- 172.21.15.64:0/1586308354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecfc102320 msgr2=0x7fecfc19f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01f44640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecfc102320 0x7fecfc19f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01f44640 1 -- 172.21.15.64:0/1586308354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecfc101960 msgr2=0x7fecfc1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01f44640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecfc101960 0x7fecfc1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01f44640 1 -- 172.21.15.64:0/1586308354 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecfc19f780 con 0x7fecfc100770 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed01f44640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc197b60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fecf8031440 tx=0x7fecf8032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf803f030 con 0x7fecfc100770 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecf8003ce0 con 0x7fecfc100770 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecfc19fa10 con 0x7fecfc100770 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8035070 con 0x7fecfc100770 2024-08-19T01:33:16.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.474+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecfc19ff50 con 0x7fecfc100770 2024-08-19T01:33:16.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.475+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecfc107ae0 con 0x7fecfc100770 2024-08-19T01:33:16.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.477+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fecf8042030 con 0x7fecfc100770 2024-08-19T01:33:16.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.478+0000 7fecf2ffd640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fece80a2ac0 0x7fece80a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.478+0000 7fed01743640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fece80a2ac0 0x7fece80a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.478+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fecf80ef690 con 0x7fecfc100770 2024-08-19T01:33:16.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.478+0000 7fed01743640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fece80a2ac0 0x7fece80a4f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fecfc199070 tx=0x7fecec00b410 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:16.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.482+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fecf80bbdc0 con 0x7fecfc100770 2024-08-19T01:33:16.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.552+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fecf80b8430 con 0x7fecfc100770 2024-08-19T01:33:16.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.694+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fecfc0624c0 con 0x7fece80a2ac0 2024-08-19T01:33:16.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.753+0000 7fecf2ffd640 1 -- 172.21.15.64:0/1586308354 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fecfc0624c0 con 0x7fece80a2ac0 2024-08-19T01:33:16.756 INFO:teuthology.orchestra.run.smithi064.stdout:Retention added to path / 2024-08-19T01:33:16.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fece80a2ac0 msgr2=0x7fece80a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fece80a2ac0 0x7fece80a4f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fecfc199070 tx=0x7fecec00b410 comp rx=0 tx=0).stop 2024-08-19T01:33:16.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 msgr2=0x7fecfc197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc197b60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fecf8031440 tx=0x7fecf8032e20 comp rx=0 tx=0).stop 2024-08-19T01:33:16.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 shutdown_connections 2024-08-19T01:33:16.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fece80a2ac0 0x7fece80a4f80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecfc102320 0x7fecfc19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecfc101960 0x7fecfc1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 --2- 172.21.15.64:0/1586308354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecfc100770 0x7fecfc197b60 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 >> 172.21.15.64:0/1586308354 conn(0x7fecfc0fbf10 msgr2=0x7fecfc104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:16.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 shutdown_connections 2024-08-19T01:33:16.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.755+0000 7fed041a8640 1 -- 172.21.15.64:0/1586308354 wait complete. 2024-08-19T01:33:16.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-08-19T01:33:16.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 -- 172.21.15.64:0/2808064116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101d50 msgr2=0x7fa5d4102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/2808064116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101d50 0x7fa5d4102150 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009900 tx=0x7fa5c402a480 comp rx=0 tx=0).stop 2024-08-19T01:33:16.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 -- 172.21.15.64:0/2808064116 shutdown_connections 2024-08-19T01:33:16.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/2808064116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5d4101360 0x7fa5d4108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/2808064116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5d41009a0 0x7fa5d4100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/2808064116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101d50 0x7fa5d4102150 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 -- 172.21.15.64:0/2808064116 >> 172.21.15.64:0/2808064116 conn(0x7fa5d40fb660 msgr2=0x7fa5d40fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:16.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 -- 172.21.15.64:0/2808064116 shutdown_connections 2024-08-19T01:33:16.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.876+0000 7fa5da1bf640 1 -- 172.21.15.64:0/2808064116 wait complete. 2024-08-19T01:33:16.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 Processor -- start 2024-08-19T01:33:16.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 -- start start 2024-08-19T01:33:16.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5d41009a0 0x7fa5d4199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101360 0x7fa5d419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5d4101d50 0x7fa5d41a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d410b300 con 0x7fa5d4101360 2024-08-19T01:33:16.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d410b180 con 0x7fa5d41009a0 2024-08-19T01:33:16.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5da1bf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d410b480 con 0x7fa5d4101d50 2024-08-19T01:33:16.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5d2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101360 0x7fa5d419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5d37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5d41009a0 0x7fa5d4199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5d2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101360 0x7fa5d419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35520/0 (socket says 172.21.15.64:35520) 2024-08-19T01:33:16.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5d4101d50 0x7fa5d41a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.877+0000 7fa5d2ffd640 1 -- 172.21.15.64:0/1772219273 learned_addr learned my addr 172.21.15.64:0/1772219273 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:16.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d3fff640 1 -- 172.21.15.64:0/1772219273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5d41009a0 msgr2=0x7fa5d4199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d3fff640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5d41009a0 0x7fa5d4199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d3fff640 1 -- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101360 msgr2=0x7fa5d419a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:16.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d3fff640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101360 0x7fa5d419a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:16.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d3fff640 1 -- 172.21.15.64:0/1772219273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d41a1980 con 0x7fa5d4101d50 2024-08-19T01:33:16.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d3fff640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5d4101d50 0x7fa5d41a1310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00da10 tx=0x7fa5cc00ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:16.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d0ff9640 1 -- 172.21.15.64:0/1772219273 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00bb70 con 0x7fa5d4101d50 2024-08-19T01:33:16.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d41a1c70 con 0x7fa5d4101d50 2024-08-19T01:33:16.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d0ff9640 1 -- 172.21.15.64:0/1772219273 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5cc00bd10 con 0x7fa5d4101d50 2024-08-19T01:33:16.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5d0ff9640 1 -- 172.21.15.64:0/1772219273 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc005020 con 0x7fa5d4101d50 2024-08-19T01:33:16.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.878+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d41a21c0 con 0x7fa5d4101d50 2024-08-19T01:33:16.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.880+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d4107b00 con 0x7fa5d4101d50 2024-08-19T01:33:16.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.880+0000 7fa5d0ff9640 1 -- 172.21.15.64:0/1772219273 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fa5cc004a40 con 0x7fa5d4101d50 2024-08-19T01:33:16.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.881+0000 7fa5d0ff9640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa5bc0a2e40 0x7fa5bc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:16.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.881+0000 7fa5d37fe640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa5bc0a2e40 0x7fa5bc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:16.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.881+0000 7fa5d0ff9640 1 -- 172.21.15.64:0/1772219273 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa5cc0cf4a0 con 0x7fa5d4101d50 2024-08-19T01:33:16.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.881+0000 7fa5d37fe640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa5bc0a2e40 0x7fa5bc0a5300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4005ab0 tx=0x7fa5c4000bc0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:16.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:16.885+0000 7fa5d0ff9640 1 -- 172.21.15.64:0/1772219273 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa5cc09b4f0 con 0x7fa5d4101d50 2024-08-19T01:33:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.086+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa5d4062360 con 0x7fa5bc0a2e40 2024-08-19T01:33:17.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.094+0000 7fa5d0ff9640 1 -- 172.21.15.64:0/1772219273 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fa5d4062360 con 0x7fa5bc0a2e40 2024-08-19T01:33:17.096 INFO:teuthology.orchestra.run.smithi064.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-08-19T00:00:00", "created": "2024-08-19T01:33:16", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-08-19T01:33:17.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa5bc0a2e40 msgr2=0x7fa5bc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:17.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa5bc0a2e40 0x7fa5bc0a5300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4005ab0 tx=0x7fa5c4000bc0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5d4101d50 msgr2=0x7fa5d41a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5d4101d50 0x7fa5d41a1310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00da10 tx=0x7fa5cc00ded0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 shutdown_connections 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa5bc0a2e40 0x7fa5bc0a5300 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5d4101d50 0x7fa5d41a1310 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4101360 0x7fa5d419a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 --2- 172.21.15.64:0/1772219273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5d41009a0 0x7fa5d4199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 >> 172.21.15.64:0/1772219273 conn(0x7fa5d40fb660 msgr2=0x7fa5d4104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 shutdown_connections 2024-08-19T01:33:17.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.096+0000 7fa5da1bf640 1 -- 172.21.15.64:0/1772219273 wait complete. 2024-08-19T01:33:17.109 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:33:17.109 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.109 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2481614044' entity='client.admin' 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.110 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-08-19T01:33:17.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: Unable to set osd_memory_target on smithi161 to 184597504: error parsing value: Value '184597504' is below minimum 939524096 2024-08-19T01:33:17.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-08-19T01:33:17.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.115 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: Unable to set osd_memory_target on smithi123 to 184594636: error parsing value: Value '184594636' is below minimum 939524096 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.116 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:33:17.117 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:16 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:33:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:33:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2481614044' entity='client.admin' 2024-08-19T01:33:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: Unable to set osd_memory_target on smithi161 to 184597504: error parsing value: Value '184597504' is below minimum 939524096 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: Unable to set osd_memory_target on smithi123 to 184594636: error parsing value: Value '184594636' is below minimum 939524096 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:17.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:33:17.169 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:16 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:33:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:33:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2481614044' entity='client.admin' 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-08-19T01:33:17.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: Unable to set osd_memory_target on smithi161 to 184597504: error parsing value: Value '184597504' is below minimum 939524096 2024-08-19T01:33:17.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.202 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: Unable to set osd_memory_target on smithi123 to 184594636: error parsing value: Value '184594636' is below minimum 939524096 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.204 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.204 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.204 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.204 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:33:17.204 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:33:17.204 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:16 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:33:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 -- 172.21.15.64:0/1400114560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 msgr2=0x7fe6140fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/1400114560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe6140fff40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe604009900 tx=0x7fe60402a410 comp rx=0 tx=0).stop 2024-08-19T01:33:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 -- 172.21.15.64:0/1400114560 shutdown_connections 2024-08-19T01:33:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/1400114560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6141034d0 0x7fe614109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/1400114560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe614102b10 0x7fe614102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/1400114560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe6140fff40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 -- 172.21.15.64:0/1400114560 >> 172.21.15.64:0/1400114560 conn(0x7fe6140fb660 msgr2=0x7fe6140fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 -- 172.21.15.64:0/1400114560 shutdown_connections 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.219+0000 7fe61a3c6640 1 -- 172.21.15.64:0/1400114560 wait complete. 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 Processor -- start 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 -- start start 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe614199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe614102b10 0x7fe61419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6141034d0 0x7fe6141a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe61410d510 con 0x7fe6141034d0 2024-08-19T01:33:17.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe61410d390 con 0x7fe614102b10 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe61a3c6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe61410d690 con 0x7fe6140ffb40 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe613fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe614199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe613fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe614199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:53066/0 (socket says 172.21.15.64:53066) 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe618963640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6141034d0 0x7fe6141a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe613fff640 1 -- 172.21.15.64:0/850163000 learned_addr learned my addr 172.21.15.64:0/850163000 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.220+0000 7fe6137fe640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe614102b10 0x7fe61419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe613fff640 1 -- 172.21.15.64:0/850163000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe614102b10 msgr2=0x7fe61419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe613fff640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe614102b10 0x7fe61419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe613fff640 1 -- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6141034d0 msgr2=0x7fe6141a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe613fff640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6141034d0 0x7fe6141a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe613fff640 1 -- 172.21.15.64:0/850163000 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6141a19c0 con 0x7fe6140ffb40 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe613fff640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe614199db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe604031610 tx=0x7fe604004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe6117fa640 1 -- 172.21.15.64:0/850163000 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60403f030 con 0x7fe6140ffb40 2024-08-19T01:33:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe6117fa640 1 -- 172.21.15.64:0/850163000 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe604003ad0 con 0x7fe6140ffb40 2024-08-19T01:33:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6141a1c50 con 0x7fe6140ffb40 2024-08-19T01:33:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe6117fa640 1 -- 172.21.15.64:0/850163000 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604035070 con 0x7fe6140ffb40 2024-08-19T01:33:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.221+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6141a2140 con 0x7fe6140ffb40 2024-08-19T01:33:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.222+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6141a2310 con 0x7fe6140ffb40 2024-08-19T01:33:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.223+0000 7fe6117fa640 1 -- 172.21.15.64:0/850163000 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fe604042030 con 0x7fe6140ffb40 2024-08-19T01:33:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.224+0000 7fe6117fa640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5fc0a2e40 0x7fe5fc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.224+0000 7fe6137fe640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5fc0a2e40 0x7fe5fc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.224+0000 7fe6117fa640 1 -- 172.21.15.64:0/850163000 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe6040f0330 con 0x7fe6140ffb40 2024-08-19T01:33:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.225+0000 7fe6137fe640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5fc0a2e40 0x7fe5fc0a5300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe608009bd0 tx=0x7fe608005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:17.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.227+0000 7fe6117fa640 1 -- 172.21.15.64:0/850163000 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe6040bc380 con 0x7fe6140ffb40 2024-08-19T01:33:17.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.430+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe614109a00 con 0x7fe5fc0a2e40 2024-08-19T01:33:17.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.437+0000 7fe6117fa640 1 -- 172.21.15.64:0/850163000 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe614109a00 con 0x7fe5fc0a2e40 2024-08-19T01:33:17.439 INFO:teuthology.orchestra.run.smithi064.stdout:/ 1m 6m3h 2024-08-19T01:33:17.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5fc0a2e40 msgr2=0x7fe5fc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:17.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5fc0a2e40 0x7fe5fc0a5300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe608009bd0 tx=0x7fe608005f90 comp rx=0 tx=0).stop 2024-08-19T01:33:17.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 msgr2=0x7fe614199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:17.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe614199db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe604031610 tx=0x7fe604004270 comp rx=0 tx=0).stop 2024-08-19T01:33:17.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 shutdown_connections 2024-08-19T01:33:17.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5fc0a2e40 0x7fe5fc0a5300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6141034d0 0x7fe6141a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe614102b10 0x7fe61419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 --2- 172.21.15.64:0/850163000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6140ffb40 0x7fe614199db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:17.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 >> 172.21.15.64:0/850163000 conn(0x7fe6140fb660 msgr2=0x7fe6141082e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:17.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 shutdown_connections 2024-08-19T01:33:17.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:17.439+0000 7fe61a3c6640 1 -- 172.21.15.64:0/850163000 wait complete. 2024-08-19T01:33:17.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-08-19T01:33:17.478 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-08-19T01:33:17.488 INFO:teuthology.run_tasks:Running task quiescer... 2024-08-19T01:33:17.530 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-19T01:33:17.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T01:33:17.799 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:18.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi064 to 176.0M 2024-08-19T01:33:18.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Unable to set osd_memory_target on smithi064 to 184603955: error parsing value: Value '184603955' is below minimum 939524096 2024-08-19T01:33:18.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi064:/etc/ceph/ceph.conf 2024-08-19T01:33:18.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi123:/etc/ceph/ceph.conf 2024-08-19T01:33:18.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:17 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi064 to 176.0M 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Unable to set osd_memory_target on smithi064 to 184603955: error parsing value: Value '184603955' is below minimum 939524096 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi064:/etc/ceph/ceph.conf 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi123:/etc/ceph/ceph.conf 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:17 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi064 to 176.0M 2024-08-19T01:33:18.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Unable to set osd_memory_target on smithi064 to 184603955: error parsing value: Value '184603955' is below minimum 939524096 2024-08-19T01:33:18.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi064:/etc/ceph/ceph.conf 2024-08-19T01:33:18.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi123:/etc/ceph/ceph.conf 2024-08-19T01:33:18.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi161:/etc/ceph/ceph.conf 2024-08-19T01:33:18.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.213 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: Updating smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.214 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.215 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:17 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:18.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 -- 172.21.15.64:0/3039029334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03141078b0 msgr2=0x7f0314107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:18.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 --2- 172.21.15.64:0/3039029334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03141078b0 0x7f0314107c90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0304009930 tx=0x7f030402a420 comp rx=0 tx=0).stop 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 -- 172.21.15.64:0/3039029334 shutdown_connections 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 --2- 172.21.15.64:0/3039029334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0314102250 0x7f031410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 --2- 172.21.15.64:0/3039029334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03141018b0 0x7f0314101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 --2- 172.21.15.64:0/3039029334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03141078b0 0x7f0314107c90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 -- 172.21.15.64:0/3039029334 >> 172.21.15.64:0/3039029334 conn(0x7f03140fd5f0 msgr2=0x7f03140ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 -- 172.21.15.64:0/3039029334 shutdown_connections 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 -- 172.21.15.64:0/3039029334 wait complete. 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.292+0000 7f0319c32640 1 Processor -- start 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0319c32640 1 -- start start 2024-08-19T01:33:18.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0319c32640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03141018b0 0x7f0314074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0319c32640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0314102250 0x7f0314074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0319c32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03141078b0 0x7f0314074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0319c32640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0314111cb0 con 0x7f0314102250 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0319c32640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0314111b30 con 0x7f03141078b0 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0319c32640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0314111e30 con 0x7f03141018b0 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0314102250 0x7f0314074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0314102250 0x7f0314074760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35556/0 (socket says 172.21.15.64:35556) 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 -- 172.21.15.64:0/2168879850 learned_addr learned my addr 172.21.15.64:0/2168879850 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0313fff640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03141078b0 0x7f0314074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f03137fe640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03141018b0 0x7f0314074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:18.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 -- 172.21.15.64:0/2168879850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03141018b0 msgr2=0x7f0314074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03141018b0 0x7f0314074220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 -- 172.21.15.64:0/2168879850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03141078b0 msgr2=0x7f0314074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03141078b0 0x7f0314074ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 -- 172.21.15.64:0/2168879850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03141a6690 con 0x7f0314102250 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.293+0000 7f0312ffd640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0314102250 0x7f0314074760 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f030800d950 tx=0x7f030800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.294+0000 7f0310ff9640 1 -- 172.21.15.64:0/2168879850 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308017070 con 0x7f0314102250 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.294+0000 7f0310ff9640 1 -- 172.21.15.64:0/2168879850 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0308009c30 con 0x7f0314102250 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.294+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03141a6980 con 0x7f0314102250 2024-08-19T01:33:18.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.294+0000 7f0310ff9640 1 -- 172.21.15.64:0/2168879850 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030800f6b0 con 0x7f0314102250 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.294+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03141a6ec0 con 0x7f0314102250 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.295+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0314101d10 con 0x7f0314102250 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.296+0000 7f0310ff9640 1 -- 172.21.15.64:0/2168879850 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f03080109d0 con 0x7f0314102250 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.296+0000 7f0310ff9640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02fc0a2d80 0x7f02fc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.297+0000 7f03137fe640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02fc0a2d80 0x7f02fc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.297+0000 7f0310ff9640 1 -- 172.21.15.64:0/2168879850 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f03080cf630 con 0x7f0314102250 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.297+0000 7f03137fe640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02fc0a2d80 0x7f02fc0a5240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0304000c00 tx=0x7f0304005a20 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:18.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.301+0000 7f0310ff9640 1 -- 172.21.15.64:0/2168879850 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f0308009de0 con 0x7f0314102250 2024-08-19T01:33:18.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.472+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0314062360 con 0x7f02fc0a2d80 2024-08-19T01:33:18.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.481+0000 7f0310ff9640 1 -- 172.21.15.64:0/2168879850 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339676 (secure 0 0 0) 0x7f0314062360 con 0x7f02fc0a2d80 2024-08-19T01:33:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:18.490 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-08-19T01:33:18.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02fc0a2d80 msgr2=0x7f02fc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:18.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02fc0a2d80 0x7f02fc0a5240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0304000c00 tx=0x7f0304005a20 comp rx=0 tx=0).stop 2024-08-19T01:33:18.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0314102250 msgr2=0x7f0314074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:18.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0314102250 0x7f0314074760 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f030800d950 tx=0x7f030800de10 comp rx=0 tx=0).stop 2024-08-19T01:33:18.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 shutdown_connections 2024-08-19T01:33:18.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02fc0a2d80 0x7f02fc0a5240 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03141078b0 0x7f0314074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0314102250 0x7f0314074760 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 --2- 172.21.15.64:0/2168879850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03141018b0 0x7f0314074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:18.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 >> 172.21.15.64:0/2168879850 conn(0x7f03140fd5f0 msgr2=0x7f03140feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:18.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 shutdown_connections 2024-08-19T01:33:18.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:18.484+0000 7f0319c32640 1 -- 172.21.15.64:0/2168879850 wait complete. 2024-08-19T01:33:19.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:18 smithi064 ceph-mon[18515]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:33:19.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:18 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:33:19.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:18 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:33:19.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:18 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:33:19.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:18 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:19.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:18 smithi064 ceph-mon[18515]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:19.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:18 smithi064 ceph-mon[18515]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-08-19T01:33:19.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:18 smithi123 ceph-mon[21829]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:33:19.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:18 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:33:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:18 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:33:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:18 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:33:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:18 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:18 smithi123 ceph-mon[21829]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:18 smithi123 ceph-mon[21829]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-08-19T01:33:19.190 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-08-19T01:33:17.832864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190430,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":863,"num_read_kb":591,"num_write":481,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1185316,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361952,"kb_used_data":12768,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579872,"statfs":{"total":1151940427776,"available":1151569788928,"internally_reserved":0,"allocated":13074432,"data_stored":9510612,"data_compressed":542357,"data_compressed_allocated":733184,"data_compressed_original":3866624,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991225"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916960+0000","last_change":"2024-08-19T01:30:49.539512+0000","last_active":"2024-08-19T01:33:13.916960+0000","last_peered":"2024-08-19T01:33:13.916960+0000","last_clean":"2024-08-19T01:33:13.916960+0000","last_became_active":"2024-08-19T01:30:49.539163+0000","last_became_peered":"2024-08-19T01:30:49.539163+0000","last_unstale":"2024-08-19T01:33:13.916960+0000","last_undegraded":"2024-08-19T01:33:13.916960+0000","last_fullsized":"2024-08-19T01:33:13.916960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:23.874916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821250+0000","last_change":"2024-08-19T01:30:51.561135+0000","last_active":"2024-08-19T01:33:13.821250+0000","last_peered":"2024-08-19T01:33:13.821250+0000","last_clean":"2024-08-19T01:33:13.821250+0000","last_became_active":"2024-08-19T01:30:51.560822+0000","last_became_peered":"2024-08-19T01:30:51.560822+0000","last_unstale":"2024-08-19T01:33:13.821250+0000","last_undegraded":"2024-08-19T01:33:13.821250+0000","last_fullsized":"2024-08-19T01:33:13.821250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:08:35.918895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821530+0000","last_change":"2024-08-19T01:30:58.595727+0000","last_active":"2024-08-19T01:33:13.821530+0000","last_peered":"2024-08-19T01:33:13.821530+0000","last_clean":"2024-08-19T01:33:13.821530+0000","last_became_active":"2024-08-19T01:30:58.595286+0000","last_became_peered":"2024-08-19T01:30:58.595286+0000","last_unstale":"2024-08-19T01:33:13.821530+0000","last_undegraded":"2024-08-19T01:33:13.821530+0000","last_fullsized":"2024-08-19T01:33:13.821530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:25:02.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821435+0000","last_change":"2024-08-19T01:30:49.575693+0000","last_active":"2024-08-19T01:33:13.821435+0000","last_peered":"2024-08-19T01:33:13.821435+0000","last_clean":"2024-08-19T01:33:13.821435+0000","last_became_active":"2024-08-19T01:30:49.575331+0000","last_became_peered":"2024-08-19T01:30:49.575331+0000","last_unstale":"2024-08-19T01:33:13.821435+0000","last_undegraded":"2024-08-19T01:33:13.821435+0000","last_fullsized":"2024-08-19T01:33:13.821435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:31:50.773160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011817+0000","last_change":"2024-08-19T01:30:51.562446+0000","last_active":"2024-08-19T01:33:14.011817+0000","last_peered":"2024-08-19T01:33:14.011817+0000","last_clean":"2024-08-19T01:33:14.011817+0000","last_became_active":"2024-08-19T01:30:51.561475+0000","last_became_peered":"2024-08-19T01:30:51.561475+0000","last_unstale":"2024-08-19T01:33:14.011817+0000","last_undegraded":"2024-08-19T01:33:14.011817+0000","last_fullsized":"2024-08-19T01:33:14.011817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:51:41.531247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821367+0000","last_change":"2024-08-19T01:30:58.591371+0000","last_active":"2024-08-19T01:33:13.821367+0000","last_peered":"2024-08-19T01:33:13.821367+0000","last_clean":"2024-08-19T01:33:13.821367+0000","last_became_active":"2024-08-19T01:30:58.590936+0000","last_became_peered":"2024-08-19T01:30:58.590936+0000","last_unstale":"2024-08-19T01:33:13.821367+0000","last_undegraded":"2024-08-19T01:33:13.821367+0000","last_fullsized":"2024-08-19T01:33:13.821367+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:07:33.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009287+0000","last_change":"2024-08-19T01:31:35.613145+0000","last_active":"2024-08-19T01:33:14.009287+0000","last_peered":"2024-08-19T01:33:14.009287+0000","last_clean":"2024-08-19T01:33:14.009287+0000","last_became_active":"2024-08-19T01:31:35.605087+0000","last_became_peered":"2024-08-19T01:31:35.605087+0000","last_unstale":"2024-08-19T01:33:14.009287+0000","last_undegraded":"2024-08-19T01:33:14.009287+0000","last_fullsized":"2024-08-19T01:33:14.009287+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:25:59.109707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009849+0000","last_change":"2024-08-19T01:30:51.555256+0000","last_active":"2024-08-19T01:33:14.009849+0000","last_peered":"2024-08-19T01:33:14.009849+0000","last_clean":"2024-08-19T01:33:14.009849+0000","last_became_active":"2024-08-19T01:30:51.554030+0000","last_became_peered":"2024-08-19T01:30:51.554030+0000","last_unstale":"2024-08-19T01:33:14.009849+0000","last_undegraded":"2024-08-19T01:33:14.009849+0000","last_fullsized":"2024-08-19T01:33:14.009849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:12:04.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821647+0000","last_change":"2024-08-19T01:30:58.589362+0000","last_active":"2024-08-19T01:33:13.821647+0000","last_peered":"2024-08-19T01:33:13.821647+0000","last_clean":"2024-08-19T01:33:13.821647+0000","last_became_active":"2024-08-19T01:30:58.588937+0000","last_became_peered":"2024-08-19T01:30:58.588937+0000","last_unstale":"2024-08-19T01:33:13.821647+0000","last_undegraded":"2024-08-19T01:33:13.821647+0000","last_fullsized":"2024-08-19T01:33:13.821647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:31.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920588+0000","last_change":"2024-08-19T01:30:49.538597+0000","last_active":"2024-08-19T01:33:13.920588+0000","last_peered":"2024-08-19T01:33:13.920588+0000","last_clean":"2024-08-19T01:33:13.920588+0000","last_became_active":"2024-08-19T01:30:49.538228+0000","last_became_peered":"2024-08-19T01:30:49.538228+0000","last_unstale":"2024-08-19T01:33:13.920588+0000","last_undegraded":"2024-08-19T01:33:13.920588+0000","last_fullsized":"2024-08-19T01:33:13.920588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:31.350962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929475+0000","last_change":"2024-08-19T01:30:51.562829+0000","last_active":"2024-08-19T01:33:13.929475+0000","last_peered":"2024-08-19T01:33:13.929475+0000","last_clean":"2024-08-19T01:33:13.929475+0000","last_became_active":"2024-08-19T01:30:51.562144+0000","last_became_peered":"2024-08-19T01:30:51.562144+0000","last_unstale":"2024-08-19T01:33:13.929475+0000","last_undegraded":"2024-08-19T01:33:13.929475+0000","last_fullsized":"2024-08-19T01:33:13.929475+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:17:09.184222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821477+0000","last_change":"2024-08-19T01:30:58.593178+0000","last_active":"2024-08-19T01:33:13.821477+0000","last_peered":"2024-08-19T01:33:13.821477+0000","last_clean":"2024-08-19T01:33:13.821477+0000","last_became_active":"2024-08-19T01:30:58.591247+0000","last_became_peered":"2024-08-19T01:30:58.591247+0000","last_unstale":"2024-08-19T01:33:13.821477+0000","last_undegraded":"2024-08-19T01:33:13.821477+0000","last_fullsized":"2024-08-19T01:33:13.821477+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:20:40.796135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009302+0000","last_change":"2024-08-19T01:30:51.562307+0000","last_active":"2024-08-19T01:33:14.009302+0000","last_peered":"2024-08-19T01:33:14.009302+0000","last_clean":"2024-08-19T01:33:14.009302+0000","last_became_active":"2024-08-19T01:30:51.561957+0000","last_became_peered":"2024-08-19T01:30:51.561957+0000","last_unstale":"2024-08-19T01:33:14.009302+0000","last_undegraded":"2024-08-19T01:33:14.009302+0000","last_fullsized":"2024-08-19T01:33:14.009302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:22.902317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009176+0000","last_change":"2024-08-19T01:31:35.609612+0000","last_active":"2024-08-19T01:33:14.009176+0000","last_peered":"2024-08-19T01:33:14.009176+0000","last_clean":"2024-08-19T01:33:14.009176+0000","last_became_active":"2024-08-19T01:31:35.604969+0000","last_became_peered":"2024-08-19T01:31:35.604969+0000","last_unstale":"2024-08-19T01:33:14.009176+0000","last_undegraded":"2024-08-19T01:33:14.009176+0000","last_fullsized":"2024-08-19T01:33:14.009176+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:59:38.089119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821743+0000","last_change":"2024-08-19T01:30:58.588984+0000","last_active":"2024-08-19T01:33:13.821743+0000","last_peered":"2024-08-19T01:33:13.821743+0000","last_clean":"2024-08-19T01:33:13.821743+0000","last_became_active":"2024-08-19T01:30:58.588504+0000","last_became_peered":"2024-08-19T01:30:58.588504+0000","last_unstale":"2024-08-19T01:33:13.821743+0000","last_undegraded":"2024-08-19T01:33:13.821743+0000","last_fullsized":"2024-08-19T01:33:13.821743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:29:14.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820715+0000","last_change":"2024-08-19T01:30:51.553293+0000","last_active":"2024-08-19T01:33:13.820715+0000","last_peered":"2024-08-19T01:33:13.820715+0000","last_clean":"2024-08-19T01:33:13.820715+0000","last_became_active":"2024-08-19T01:30:51.552990+0000","last_became_peered":"2024-08-19T01:30:51.552990+0000","last_unstale":"2024-08-19T01:33:13.820715+0000","last_undegraded":"2024-08-19T01:33:13.820715+0000","last_fullsized":"2024-08-19T01:33:13.820715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:28:33.193014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821634+0000","last_change":"2024-08-19T01:30:49.537742+0000","last_active":"2024-08-19T01:33:13.821634+0000","last_peered":"2024-08-19T01:33:13.821634+0000","last_clean":"2024-08-19T01:33:13.821634+0000","last_became_active":"2024-08-19T01:30:49.537153+0000","last_became_peered":"2024-08-19T01:30:49.537153+0000","last_unstale":"2024-08-19T01:33:13.821634+0000","last_undegraded":"2024-08-19T01:33:13.821634+0000","last_fullsized":"2024-08-19T01:33:13.821634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:25:03.650900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821745+0000","last_change":"2024-08-19T01:32:35.668059+0000","last_active":"2024-08-19T01:33:13.821745+0000","last_peered":"2024-08-19T01:33:13.821745+0000","last_clean":"2024-08-19T01:33:13.821745+0000","last_became_active":"2024-08-19T01:32:35.663768+0000","last_became_peered":"2024-08-19T01:32:35.663768+0000","last_unstale":"2024-08-19T01:33:13.821745+0000","last_undegraded":"2024-08-19T01:33:13.821745+0000","last_fullsized":"2024-08-19T01:33:13.821745+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:19:38.669783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009175+0000","last_change":"2024-08-19T01:30:51.552126+0000","last_active":"2024-08-19T01:33:14.009175+0000","last_peered":"2024-08-19T01:33:14.009175+0000","last_clean":"2024-08-19T01:33:14.009175+0000","last_became_active":"2024-08-19T01:30:51.551795+0000","last_became_peered":"2024-08-19T01:30:51.551795+0000","last_unstale":"2024-08-19T01:33:14.009175+0000","last_undegraded":"2024-08-19T01:33:14.009175+0000","last_fullsized":"2024-08-19T01:33:14.009175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:22:54.705359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820641+0000","last_change":"2024-08-19T01:30:49.536659+0000","last_active":"2024-08-19T01:33:13.820641+0000","last_peered":"2024-08-19T01:33:13.820641+0000","last_clean":"2024-08-19T01:33:13.820641+0000","last_became_active":"2024-08-19T01:30:49.536366+0000","last_became_peered":"2024-08-19T01:30:49.536366+0000","last_unstale":"2024-08-19T01:33:13.820641+0000","last_undegraded":"2024-08-19T01:33:13.820641+0000","last_fullsized":"2024-08-19T01:33:13.820641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:58:52.547461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916618+0000","last_change":"2024-08-19T01:30:58.593724+0000","last_active":"2024-08-19T01:33:13.916618+0000","last_peered":"2024-08-19T01:33:13.916618+0000","last_clean":"2024-08-19T01:33:13.916618+0000","last_became_active":"2024-08-19T01:30:58.593383+0000","last_became_peered":"2024-08-19T01:30:58.593383+0000","last_unstale":"2024-08-19T01:33:13.916618+0000","last_undegraded":"2024-08-19T01:33:13.916618+0000","last_fullsized":"2024-08-19T01:33:13.916618+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:54:26.424139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920243+0000","last_change":"2024-08-19T01:30:51.557682+0000","last_active":"2024-08-19T01:33:13.920243+0000","last_peered":"2024-08-19T01:33:13.920243+0000","last_clean":"2024-08-19T01:33:13.920243+0000","last_became_active":"2024-08-19T01:30:51.557351+0000","last_became_peered":"2024-08-19T01:30:51.557351+0000","last_unstale":"2024-08-19T01:33:13.920243+0000","last_undegraded":"2024-08-19T01:33:13.920243+0000","last_fullsized":"2024-08-19T01:33:13.920243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:41.484955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009059+0000","last_change":"2024-08-19T01:30:49.575996+0000","last_active":"2024-08-19T01:33:14.009059+0000","last_peered":"2024-08-19T01:33:14.009059+0000","last_clean":"2024-08-19T01:33:14.009059+0000","last_became_active":"2024-08-19T01:30:49.575607+0000","last_became_peered":"2024-08-19T01:30:49.575607+0000","last_unstale":"2024-08-19T01:33:14.009059+0000","last_undegraded":"2024-08-19T01:33:14.009059+0000","last_fullsized":"2024-08-19T01:33:14.009059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:20:33.467100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821854+0000","last_change":"2024-08-19T01:30:58.592629+0000","last_active":"2024-08-19T01:33:13.821854+0000","last_peered":"2024-08-19T01:33:13.821854+0000","last_clean":"2024-08-19T01:33:13.821854+0000","last_became_active":"2024-08-19T01:30:58.592122+0000","last_became_peered":"2024-08-19T01:30:58.592122+0000","last_unstale":"2024-08-19T01:33:13.821854+0000","last_undegraded":"2024-08-19T01:33:13.821854+0000","last_fullsized":"2024-08-19T01:33:13.821854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:19:09.846829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011482+0000","last_change":"2024-08-19T01:31:35.746416+0000","last_active":"2024-08-19T01:33:14.011482+0000","last_peered":"2024-08-19T01:33:14.011482+0000","last_clean":"2024-08-19T01:33:14.011482+0000","last_became_active":"2024-08-19T01:31:35.746110+0000","last_became_peered":"2024-08-19T01:31:35.746110+0000","last_unstale":"2024-08-19T01:33:14.011482+0000","last_undegraded":"2024-08-19T01:33:14.011482+0000","last_fullsized":"2024-08-19T01:33:14.011482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:29:29.789785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821949+0000","last_change":"2024-08-19T01:30:49.711366+0000","last_active":"2024-08-19T01:33:13.821949+0000","last_peered":"2024-08-19T01:33:13.821949+0000","last_clean":"2024-08-19T01:33:13.821949+0000","last_became_active":"2024-08-19T01:30:49.711078+0000","last_became_peered":"2024-08-19T01:30:49.711078+0000","last_unstale":"2024-08-19T01:33:13.821949+0000","last_undegraded":"2024-08-19T01:33:13.821949+0000","last_fullsized":"2024-08-19T01:33:13.821949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:11:16.767867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821530+0000","last_change":"2024-08-19T01:30:58.591845+0000","last_active":"2024-08-19T01:33:13.821530+0000","last_peered":"2024-08-19T01:33:13.821530+0000","last_clean":"2024-08-19T01:33:13.821530+0000","last_became_active":"2024-08-19T01:30:58.591453+0000","last_became_peered":"2024-08-19T01:30:58.591453+0000","last_unstale":"2024-08-19T01:33:13.821530+0000","last_undegraded":"2024-08-19T01:33:13.821530+0000","last_fullsized":"2024-08-19T01:33:13.821530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:37:09.521514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820925+0000","last_change":"2024-08-19T01:30:51.563156+0000","last_active":"2024-08-19T01:33:13.820925+0000","last_peered":"2024-08-19T01:33:13.820925+0000","last_clean":"2024-08-19T01:33:13.820925+0000","last_became_active":"2024-08-19T01:30:51.562860+0000","last_became_peered":"2024-08-19T01:30:51.562860+0000","last_unstale":"2024-08-19T01:33:13.820925+0000","last_undegraded":"2024-08-19T01:33:13.820925+0000","last_fullsized":"2024-08-19T01:33:13.820925+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:54:11.768833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":112,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.053205+0000","last_change":"2024-08-19T01:30:49.536648+0000","last_active":"2024-08-19T01:33:14.053205+0000","last_peered":"2024-08-19T01:33:14.053205+0000","last_clean":"2024-08-19T01:33:14.053205+0000","last_became_active":"2024-08-19T01:30:49.536080+0000","last_became_peered":"2024-08-19T01:30:49.536080+0000","last_unstale":"2024-08-19T01:33:14.053205+0000","last_undegraded":"2024-08-19T01:33:14.053205+0000","last_fullsized":"2024-08-19T01:33:14.053205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:03:46.804252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":87,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820301+0000","last_change":"2024-08-19T01:30:58.592253+0000","last_active":"2024-08-19T01:33:13.820301+0000","last_peered":"2024-08-19T01:33:13.820301+0000","last_clean":"2024-08-19T01:33:13.820301+0000","last_became_active":"2024-08-19T01:30:58.591911+0000","last_became_peered":"2024-08-19T01:30:58.591911+0000","last_unstale":"2024-08-19T01:33:13.820301+0000","last_undegraded":"2024-08-19T01:33:13.820301+0000","last_fullsized":"2024-08-19T01:33:13.820301+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:02:27.112597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010224+0000","last_change":"2024-08-19T01:30:51.547607+0000","last_active":"2024-08-19T01:33:14.010224+0000","last_peered":"2024-08-19T01:33:14.010224+0000","last_clean":"2024-08-19T01:33:14.010224+0000","last_became_active":"2024-08-19T01:30:51.547316+0000","last_became_peered":"2024-08-19T01:30:51.547316+0000","last_unstale":"2024-08-19T01:33:14.010224+0000","last_undegraded":"2024-08-19T01:33:14.010224+0000","last_fullsized":"2024-08-19T01:33:14.010224+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:48:58.300508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920697+0000","last_change":"2024-08-19T01:30:49.576942+0000","last_active":"2024-08-19T01:33:13.920697+0000","last_peered":"2024-08-19T01:33:13.920697+0000","last_clean":"2024-08-19T01:33:13.920697+0000","last_became_active":"2024-08-19T01:30:49.576528+0000","last_became_peered":"2024-08-19T01:30:49.576528+0000","last_unstale":"2024-08-19T01:33:13.920697+0000","last_undegraded":"2024-08-19T01:33:13.920697+0000","last_fullsized":"2024-08-19T01:33:13.920697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:23:53.259395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009281+0000","last_change":"2024-08-19T01:30:58.591577+0000","last_active":"2024-08-19T01:33:14.009281+0000","last_peered":"2024-08-19T01:33:14.009281+0000","last_clean":"2024-08-19T01:33:14.009281+0000","last_became_active":"2024-08-19T01:30:58.591181+0000","last_became_peered":"2024-08-19T01:30:58.591181+0000","last_unstale":"2024-08-19T01:33:14.009281+0000","last_undegraded":"2024-08-19T01:33:14.009281+0000","last_fullsized":"2024-08-19T01:33:14.009281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:02:59.682147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932531+0000","last_change":"2024-08-19T01:30:51.548556+0000","last_active":"2024-08-19T01:33:13.932531+0000","last_peered":"2024-08-19T01:33:13.932531+0000","last_clean":"2024-08-19T01:33:13.932531+0000","last_became_active":"2024-08-19T01:30:51.547838+0000","last_became_peered":"2024-08-19T01:30:51.547838+0000","last_unstale":"2024-08-19T01:33:13.932531+0000","last_undegraded":"2024-08-19T01:33:13.932531+0000","last_fullsized":"2024-08-19T01:33:13.932531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:08:49.671964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916085+0000","last_change":"2024-08-19T01:30:49.576799+0000","last_active":"2024-08-19T01:33:13.916085+0000","last_peered":"2024-08-19T01:33:13.916085+0000","last_clean":"2024-08-19T01:33:13.916085+0000","last_became_active":"2024-08-19T01:30:49.576385+0000","last_became_peered":"2024-08-19T01:30:49.576385+0000","last_unstale":"2024-08-19T01:33:13.916085+0000","last_undegraded":"2024-08-19T01:33:13.916085+0000","last_fullsized":"2024-08-19T01:33:13.916085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:34:18.287518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920261+0000","last_change":"2024-08-19T01:30:58.593703+0000","last_active":"2024-08-19T01:33:13.920261+0000","last_peered":"2024-08-19T01:33:13.920261+0000","last_clean":"2024-08-19T01:33:13.920261+0000","last_became_active":"2024-08-19T01:30:58.593303+0000","last_became_peered":"2024-08-19T01:30:58.593303+0000","last_unstale":"2024-08-19T01:33:13.920261+0000","last_undegraded":"2024-08-19T01:33:13.920261+0000","last_fullsized":"2024-08-19T01:33:13.920261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:12:43.359262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932455+0000","last_change":"2024-08-19T01:31:35.601173+0000","last_active":"2024-08-19T01:33:13.932455+0000","last_peered":"2024-08-19T01:33:13.932455+0000","last_clean":"2024-08-19T01:33:13.932455+0000","last_became_active":"2024-08-19T01:31:35.600742+0000","last_became_peered":"2024-08-19T01:31:35.600742+0000","last_unstale":"2024-08-19T01:33:13.932455+0000","last_undegraded":"2024-08-19T01:33:13.932455+0000","last_fullsized":"2024-08-19T01:33:13.932455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:36:48.977124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009550+0000","last_change":"2024-08-19T01:30:49.540070+0000","last_active":"2024-08-19T01:33:14.009550+0000","last_peered":"2024-08-19T01:33:14.009550+0000","last_clean":"2024-08-19T01:33:14.009550+0000","last_became_active":"2024-08-19T01:30:49.539706+0000","last_became_peered":"2024-08-19T01:30:49.539706+0000","last_unstale":"2024-08-19T01:33:14.009550+0000","last_undegraded":"2024-08-19T01:33:14.009550+0000","last_fullsized":"2024-08-19T01:33:14.009550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:14:15.813670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009939+0000","last_change":"2024-08-19T01:30:58.590766+0000","last_active":"2024-08-19T01:33:14.009939+0000","last_peered":"2024-08-19T01:33:14.009939+0000","last_clean":"2024-08-19T01:33:14.009939+0000","last_became_active":"2024-08-19T01:30:58.590415+0000","last_became_peered":"2024-08-19T01:30:58.590415+0000","last_unstale":"2024-08-19T01:33:14.009939+0000","last_undegraded":"2024-08-19T01:33:14.009939+0000","last_fullsized":"2024-08-19T01:33:14.009939+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:48:59.261086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914324+0000","last_change":"2024-08-19T01:31:35.602332+0000","last_active":"2024-08-19T01:33:13.914324+0000","last_peered":"2024-08-19T01:33:13.914324+0000","last_clean":"2024-08-19T01:33:13.914324+0000","last_became_active":"2024-08-19T01:31:35.602007+0000","last_became_peered":"2024-08-19T01:31:35.602007+0000","last_unstale":"2024-08-19T01:33:13.914324+0000","last_undegraded":"2024-08-19T01:33:13.914324+0000","last_fullsized":"2024-08-19T01:33:13.914324+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:17:47.694567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":148,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009619+0000","last_change":"2024-08-19T01:30:49.575995+0000","last_active":"2024-08-19T01:33:14.009619+0000","last_peered":"2024-08-19T01:33:14.009619+0000","last_clean":"2024-08-19T01:33:14.009619+0000","last_became_active":"2024-08-19T01:30:49.575249+0000","last_became_peered":"2024-08-19T01:30:49.575249+0000","last_unstale":"2024-08-19T01:33:14.009619+0000","last_undegraded":"2024-08-19T01:33:14.009619+0000","last_fullsized":"2024-08-19T01:33:14.009619+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:39:05.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011011+0000","last_change":"2024-08-19T01:32:35.725750+0000","last_active":"2024-08-19T01:33:14.011011+0000","last_peered":"2024-08-19T01:33:14.011011+0000","last_clean":"2024-08-19T01:33:14.011011+0000","last_became_active":"2024-08-19T01:32:35.725320+0000","last_became_peered":"2024-08-19T01:32:35.725320+0000","last_unstale":"2024-08-19T01:33:14.011011+0000","last_undegraded":"2024-08-19T01:33:14.011011+0000","last_fullsized":"2024-08-19T01:33:14.011011+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:10.492801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009295+0000","last_change":"2024-08-19T01:30:51.549158+0000","last_active":"2024-08-19T01:33:14.009295+0000","last_peered":"2024-08-19T01:33:14.009295+0000","last_clean":"2024-08-19T01:33:14.009295+0000","last_became_active":"2024-08-19T01:30:51.548854+0000","last_became_peered":"2024-08-19T01:30:51.548854+0000","last_unstale":"2024-08-19T01:33:14.009295+0000","last_undegraded":"2024-08-19T01:33:14.009295+0000","last_fullsized":"2024-08-19T01:33:14.009295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:35.661758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009708+0000","last_change":"2024-08-19T01:30:49.538282+0000","last_active":"2024-08-19T01:33:14.009708+0000","last_peered":"2024-08-19T01:33:14.009708+0000","last_clean":"2024-08-19T01:33:14.009708+0000","last_became_active":"2024-08-19T01:30:49.537918+0000","last_became_peered":"2024-08-19T01:30:49.537918+0000","last_unstale":"2024-08-19T01:33:14.009708+0000","last_undegraded":"2024-08-19T01:33:14.009708+0000","last_fullsized":"2024-08-19T01:33:14.009708+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:27.886958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929499+0000","last_change":"2024-08-19T01:30:58.591153+0000","last_active":"2024-08-19T01:33:13.929499+0000","last_peered":"2024-08-19T01:33:13.929499+0000","last_clean":"2024-08-19T01:33:13.929499+0000","last_became_active":"2024-08-19T01:30:58.590738+0000","last_became_peered":"2024-08-19T01:30:58.590738+0000","last_unstale":"2024-08-19T01:33:13.929499+0000","last_undegraded":"2024-08-19T01:33:13.929499+0000","last_fullsized":"2024-08-19T01:33:13.929499+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:29.073825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820667+0000","last_change":"2024-08-19T01:30:51.554878+0000","last_active":"2024-08-19T01:33:13.820667+0000","last_peered":"2024-08-19T01:33:13.820667+0000","last_clean":"2024-08-19T01:33:13.820667+0000","last_became_active":"2024-08-19T01:30:51.553743+0000","last_became_peered":"2024-08-19T01:30:51.553743+0000","last_unstale":"2024-08-19T01:33:13.820667+0000","last_undegraded":"2024-08-19T01:33:13.820667+0000","last_fullsized":"2024-08-19T01:33:13.820667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:08:41.932641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822048+0000","last_change":"2024-08-19T01:30:49.542043+0000","last_active":"2024-08-19T01:33:13.822048+0000","last_peered":"2024-08-19T01:33:13.822048+0000","last_clean":"2024-08-19T01:33:13.822048+0000","last_became_active":"2024-08-19T01:30:49.541551+0000","last_became_peered":"2024-08-19T01:30:49.541551+0000","last_unstale":"2024-08-19T01:33:13.822048+0000","last_undegraded":"2024-08-19T01:33:13.822048+0000","last_fullsized":"2024-08-19T01:33:13.822048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:46:14.781516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":371,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009819+0000","last_change":"2024-08-19T01:29:41.574391+0000","last_active":"2024-08-19T01:33:14.009819+0000","last_peered":"2024-08-19T01:33:14.009819+0000","last_clean":"2024-08-19T01:33:14.009819+0000","last_became_active":"2024-08-19T01:29:41.573885+0000","last_became_peered":"2024-08-19T01:29:41.573885+0000","last_unstale":"2024-08-19T01:33:14.009819+0000","last_undegraded":"2024-08-19T01:33:14.009819+0000","last_fullsized":"2024-08-19T01:33:14.009819+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_clean_scrub_stamp":"2024-08-19T01:26:37.926225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:55:38.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010021+0000","last_change":"2024-08-19T01:30:58.598415+0000","last_active":"2024-08-19T01:33:14.010021+0000","last_peered":"2024-08-19T01:33:14.010021+0000","last_clean":"2024-08-19T01:33:14.010021+0000","last_became_active":"2024-08-19T01:30:58.598086+0000","last_became_peered":"2024-08-19T01:30:58.598086+0000","last_unstale":"2024-08-19T01:33:14.010021+0000","last_undegraded":"2024-08-19T01:33:14.010021+0000","last_fullsized":"2024-08-19T01:33:14.010021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:21:37.338869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010133+0000","last_change":"2024-08-19T01:30:51.560407+0000","last_active":"2024-08-19T01:33:14.010133+0000","last_peered":"2024-08-19T01:33:14.010133+0000","last_clean":"2024-08-19T01:33:14.010133+0000","last_became_active":"2024-08-19T01:30:51.559881+0000","last_became_peered":"2024-08-19T01:30:51.559881+0000","last_unstale":"2024-08-19T01:33:14.010133+0000","last_undegraded":"2024-08-19T01:33:14.010133+0000","last_fullsized":"2024-08-19T01:33:14.010133+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:24:37.928254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822063+0000","last_change":"2024-08-19T01:30:49.711171+0000","last_active":"2024-08-19T01:33:13.822063+0000","last_peered":"2024-08-19T01:33:13.822063+0000","last_clean":"2024-08-19T01:33:13.822063+0000","last_became_active":"2024-08-19T01:30:49.710707+0000","last_became_peered":"2024-08-19T01:30:49.710707+0000","last_unstale":"2024-08-19T01:33:13.822063+0000","last_undegraded":"2024-08-19T01:33:13.822063+0000","last_fullsized":"2024-08-19T01:33:13.822063+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:58:20.924502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009117+0000","last_change":"2024-08-19T01:30:58.590829+0000","last_active":"2024-08-19T01:33:14.009117+0000","last_peered":"2024-08-19T01:33:14.009117+0000","last_clean":"2024-08-19T01:33:14.009117+0000","last_became_active":"2024-08-19T01:30:58.590460+0000","last_became_peered":"2024-08-19T01:30:58.590460+0000","last_unstale":"2024-08-19T01:33:14.009117+0000","last_undegraded":"2024-08-19T01:33:14.009117+0000","last_fullsized":"2024-08-19T01:33:14.009117+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:04.118550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009951+0000","last_change":"2024-08-19T01:30:51.564990+0000","last_active":"2024-08-19T01:33:14.009951+0000","last_peered":"2024-08-19T01:33:14.009951+0000","last_clean":"2024-08-19T01:33:14.009951+0000","last_became_active":"2024-08-19T01:30:51.564653+0000","last_became_peered":"2024-08-19T01:30:51.564653+0000","last_unstale":"2024-08-19T01:33:14.009951+0000","last_undegraded":"2024-08-19T01:33:14.009951+0000","last_fullsized":"2024-08-19T01:33:14.009951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:15:41.154376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.046623+0000","last_change":"2024-08-19T01:30:49.537637+0000","last_active":"2024-08-19T01:33:14.046623+0000","last_peered":"2024-08-19T01:33:14.046623+0000","last_clean":"2024-08-19T01:33:14.046623+0000","last_became_active":"2024-08-19T01:30:49.537277+0000","last_became_peered":"2024-08-19T01:30:49.537277+0000","last_unstale":"2024-08-19T01:33:14.046623+0000","last_undegraded":"2024-08-19T01:33:14.046623+0000","last_fullsized":"2024-08-19T01:33:14.046623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:13.547597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010579+0000","last_change":"2024-08-19T01:30:58.594675+0000","last_active":"2024-08-19T01:33:14.010579+0000","last_peered":"2024-08-19T01:33:14.010579+0000","last_clean":"2024-08-19T01:33:14.010579+0000","last_became_active":"2024-08-19T01:30:58.594323+0000","last_became_peered":"2024-08-19T01:30:58.594323+0000","last_unstale":"2024-08-19T01:33:14.010579+0000","last_undegraded":"2024-08-19T01:33:14.010579+0000","last_fullsized":"2024-08-19T01:33:14.010579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:21:05.447106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010126+0000","last_change":"2024-08-19T01:30:51.565014+0000","last_active":"2024-08-19T01:33:14.010126+0000","last_peered":"2024-08-19T01:33:14.010126+0000","last_clean":"2024-08-19T01:33:14.010126+0000","last_became_active":"2024-08-19T01:30:51.564672+0000","last_became_peered":"2024-08-19T01:30:51.564672+0000","last_unstale":"2024-08-19T01:33:14.010126+0000","last_undegraded":"2024-08-19T01:33:14.010126+0000","last_fullsized":"2024-08-19T01:33:14.010126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:56:07.404042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010841+0000","last_change":"2024-08-19T01:30:49.538423+0000","last_active":"2024-08-19T01:33:14.010841+0000","last_peered":"2024-08-19T01:33:14.010841+0000","last_clean":"2024-08-19T01:33:14.010841+0000","last_became_active":"2024-08-19T01:30:49.538049+0000","last_became_peered":"2024-08-19T01:30:49.538049+0000","last_unstale":"2024-08-19T01:33:14.010841+0000","last_undegraded":"2024-08-19T01:33:14.010841+0000","last_fullsized":"2024-08-19T01:33:14.010841+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:27:41.449742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929000+0000","last_change":"2024-08-19T01:30:58.590963+0000","last_active":"2024-08-19T01:33:13.929000+0000","last_peered":"2024-08-19T01:33:13.929000+0000","last_clean":"2024-08-19T01:33:13.929000+0000","last_became_active":"2024-08-19T01:30:58.590584+0000","last_became_peered":"2024-08-19T01:30:58.590584+0000","last_unstale":"2024-08-19T01:33:13.929000+0000","last_undegraded":"2024-08-19T01:33:13.929000+0000","last_fullsized":"2024-08-19T01:33:13.929000+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:37.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009458+0000","last_change":"2024-08-19T01:30:51.561298+0000","last_active":"2024-08-19T01:33:14.009458+0000","last_peered":"2024-08-19T01:33:14.009458+0000","last_clean":"2024-08-19T01:33:14.009458+0000","last_became_active":"2024-08-19T01:30:51.560919+0000","last_became_peered":"2024-08-19T01:30:51.560919+0000","last_unstale":"2024-08-19T01:33:14.009458+0000","last_undegraded":"2024-08-19T01:33:14.009458+0000","last_fullsized":"2024-08-19T01:33:14.009458+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:04:39.808838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920821+0000","last_change":"2024-08-19T01:30:49.541968+0000","last_active":"2024-08-19T01:33:13.920821+0000","last_peered":"2024-08-19T01:33:13.920821+0000","last_clean":"2024-08-19T01:33:13.920821+0000","last_became_active":"2024-08-19T01:30:49.541512+0000","last_became_peered":"2024-08-19T01:30:49.541512+0000","last_unstale":"2024-08-19T01:33:13.920821+0000","last_undegraded":"2024-08-19T01:33:13.920821+0000","last_fullsized":"2024-08-19T01:33:13.920821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:34:28.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929302+0000","last_change":"2024-08-19T01:30:58.594776+0000","last_active":"2024-08-19T01:33:13.929302+0000","last_peered":"2024-08-19T01:33:13.929302+0000","last_clean":"2024-08-19T01:33:13.929302+0000","last_became_active":"2024-08-19T01:30:58.594414+0000","last_became_peered":"2024-08-19T01:30:58.594414+0000","last_unstale":"2024-08-19T01:33:13.929302+0000","last_undegraded":"2024-08-19T01:33:13.929302+0000","last_fullsized":"2024-08-19T01:33:13.929302+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:32:20.121504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929211+0000","last_change":"2024-08-19T01:30:51.556696+0000","last_active":"2024-08-19T01:33:13.929211+0000","last_peered":"2024-08-19T01:33:13.929211+0000","last_clean":"2024-08-19T01:33:13.929211+0000","last_became_active":"2024-08-19T01:30:51.555873+0000","last_became_peered":"2024-08-19T01:30:51.555873+0000","last_unstale":"2024-08-19T01:33:13.929211+0000","last_undegraded":"2024-08-19T01:33:13.929211+0000","last_fullsized":"2024-08-19T01:33:13.929211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:50:27.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920308+0000","last_change":"2024-08-19T01:30:49.576925+0000","last_active":"2024-08-19T01:33:13.920308+0000","last_peered":"2024-08-19T01:33:13.920308+0000","last_clean":"2024-08-19T01:33:13.920308+0000","last_became_active":"2024-08-19T01:30:49.576509+0000","last_became_peered":"2024-08-19T01:30:49.576509+0000","last_unstale":"2024-08-19T01:33:13.920308+0000","last_undegraded":"2024-08-19T01:33:13.920308+0000","last_fullsized":"2024-08-19T01:33:13.920308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:43:01.887020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010743+0000","last_change":"2024-08-19T01:32:35.826488+0000","last_active":"2024-08-19T01:33:14.010743+0000","last_peered":"2024-08-19T01:33:14.010743+0000","last_clean":"2024-08-19T01:33:14.010743+0000","last_became_active":"2024-08-19T01:32:35.826036+0000","last_became_peered":"2024-08-19T01:32:35.826036+0000","last_unstale":"2024-08-19T01:33:14.010743+0000","last_undegraded":"2024-08-19T01:33:14.010743+0000","last_fullsized":"2024-08-19T01:33:14.010743+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:16:03.388117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009567+0000","last_change":"2024-08-19T01:30:51.559622+0000","last_active":"2024-08-19T01:33:14.009567+0000","last_peered":"2024-08-19T01:33:14.009567+0000","last_clean":"2024-08-19T01:33:14.009567+0000","last_became_active":"2024-08-19T01:30:51.559263+0000","last_became_peered":"2024-08-19T01:30:51.559263+0000","last_unstale":"2024-08-19T01:33:14.009567+0000","last_undegraded":"2024-08-19T01:33:14.009567+0000","last_fullsized":"2024-08-19T01:33:14.009567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:11:25.758915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":155,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.739646+0000","last_change":"2024-08-19T01:30:49.540102+0000","last_active":"2024-08-19T01:33:14.739646+0000","last_peered":"2024-08-19T01:33:14.739646+0000","last_clean":"2024-08-19T01:33:14.739646+0000","last_became_active":"2024-08-19T01:30:49.539740+0000","last_became_peered":"2024-08-19T01:30:49.539740+0000","last_unstale":"2024-08-19T01:33:14.739646+0000","last_undegraded":"2024-08-19T01:33:14.739646+0000","last_fullsized":"2024-08-19T01:33:14.739646+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:47:21.684925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1899,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916713+0000","last_change":"2024-08-19T01:30:58.595748+0000","last_active":"2024-08-19T01:33:13.916713+0000","last_peered":"2024-08-19T01:33:13.916713+0000","last_clean":"2024-08-19T01:33:13.916713+0000","last_became_active":"2024-08-19T01:30:58.595396+0000","last_became_peered":"2024-08-19T01:30:58.595396+0000","last_unstale":"2024-08-19T01:33:13.916713+0000","last_undegraded":"2024-08-19T01:33:13.916713+0000","last_fullsized":"2024-08-19T01:33:13.916713+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:58:29.113687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009400+0000","last_change":"2024-08-19T01:30:51.559159+0000","last_active":"2024-08-19T01:33:14.009400+0000","last_peered":"2024-08-19T01:33:14.009400+0000","last_clean":"2024-08-19T01:33:14.009400+0000","last_became_active":"2024-08-19T01:30:51.558226+0000","last_became_peered":"2024-08-19T01:30:51.558226+0000","last_unstale":"2024-08-19T01:33:14.009400+0000","last_undegraded":"2024-08-19T01:33:14.009400+0000","last_fullsized":"2024-08-19T01:33:14.009400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:12:54.447301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929394+0000","last_change":"2024-08-19T01:30:49.540806+0000","last_active":"2024-08-19T01:33:13.929394+0000","last_peered":"2024-08-19T01:33:13.929394+0000","last_clean":"2024-08-19T01:33:13.929394+0000","last_became_active":"2024-08-19T01:30:49.540439+0000","last_became_peered":"2024-08-19T01:30:49.540439+0000","last_unstale":"2024-08-19T01:33:13.929394+0000","last_undegraded":"2024-08-19T01:33:13.929394+0000","last_fullsized":"2024-08-19T01:33:13.929394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:40.088842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009235+0000","last_change":"2024-08-19T01:30:58.592661+0000","last_active":"2024-08-19T01:33:14.009235+0000","last_peered":"2024-08-19T01:33:14.009235+0000","last_clean":"2024-08-19T01:33:14.009235+0000","last_became_active":"2024-08-19T01:30:58.592066+0000","last_became_peered":"2024-08-19T01:30:58.592066+0000","last_unstale":"2024-08-19T01:33:14.009235+0000","last_undegraded":"2024-08-19T01:33:14.009235+0000","last_fullsized":"2024-08-19T01:33:14.009235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:35.039682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009768+0000","last_change":"2024-08-19T01:30:51.558497+0000","last_active":"2024-08-19T01:33:14.009768+0000","last_peered":"2024-08-19T01:33:14.009768+0000","last_clean":"2024-08-19T01:33:14.009768+0000","last_became_active":"2024-08-19T01:30:51.558185+0000","last_became_peered":"2024-08-19T01:30:51.558185+0000","last_unstale":"2024-08-19T01:33:14.009768+0000","last_undegraded":"2024-08-19T01:33:14.009768+0000","last_fullsized":"2024-08-19T01:33:14.009768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:30:42.766269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821845+0000","last_change":"2024-08-19T01:30:49.539112+0000","last_active":"2024-08-19T01:33:13.821845+0000","last_peered":"2024-08-19T01:33:13.821845+0000","last_clean":"2024-08-19T01:33:13.821845+0000","last_became_active":"2024-08-19T01:30:49.538798+0000","last_became_peered":"2024-08-19T01:30:49.538798+0000","last_unstale":"2024-08-19T01:33:13.821845+0000","last_undegraded":"2024-08-19T01:33:13.821845+0000","last_fullsized":"2024-08-19T01:33:13.821845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:10:58.977041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929436+0000","last_change":"2024-08-19T01:30:58.590727+0000","last_active":"2024-08-19T01:33:13.929436+0000","last_peered":"2024-08-19T01:33:13.929436+0000","last_clean":"2024-08-19T01:33:13.929436+0000","last_became_active":"2024-08-19T01:30:58.590208+0000","last_became_peered":"2024-08-19T01:30:58.590208+0000","last_unstale":"2024-08-19T01:33:13.929436+0000","last_undegraded":"2024-08-19T01:33:13.929436+0000","last_fullsized":"2024-08-19T01:33:13.929436+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:36.422915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920646+0000","last_change":"2024-08-19T01:30:51.565113+0000","last_active":"2024-08-19T01:33:13.920646+0000","last_peered":"2024-08-19T01:33:13.920646+0000","last_clean":"2024-08-19T01:33:13.920646+0000","last_became_active":"2024-08-19T01:30:51.564769+0000","last_became_peered":"2024-08-19T01:30:51.564769+0000","last_unstale":"2024-08-19T01:33:13.920646+0000","last_undegraded":"2024-08-19T01:33:13.920646+0000","last_fullsized":"2024-08-19T01:33:13.920646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:13:28.373114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":311,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.739508+0000","last_change":"2024-08-19T01:31:35.610319+0000","last_active":"2024-08-19T01:33:14.739508+0000","last_peered":"2024-08-19T01:33:14.739508+0000","last_clean":"2024-08-19T01:33:14.739508+0000","last_became_active":"2024-08-19T01:31:35.602527+0000","last_became_peered":"2024-08-19T01:31:35.602527+0000","last_unstale":"2024-08-19T01:33:14.739508+0000","last_undegraded":"2024-08-19T01:33:14.739508+0000","last_fullsized":"2024-08-19T01:33:14.739508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:50:05.511332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68780,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":210,"num_read_kb":170,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3841,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.915346+0000","last_change":"2024-08-19T01:30:58.594836+0000","last_active":"2024-08-19T01:33:13.915346+0000","last_peered":"2024-08-19T01:33:13.915346+0000","last_clean":"2024-08-19T01:33:13.915346+0000","last_became_active":"2024-08-19T01:30:58.594441+0000","last_became_peered":"2024-08-19T01:30:58.594441+0000","last_unstale":"2024-08-19T01:33:13.915346+0000","last_undegraded":"2024-08-19T01:33:13.915346+0000","last_fullsized":"2024-08-19T01:33:13.915346+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:29:16.482181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009266+0000","last_change":"2024-08-19T01:30:51.562567+0000","last_active":"2024-08-19T01:33:14.009266+0000","last_peered":"2024-08-19T01:33:14.009266+0000","last_clean":"2024-08-19T01:33:14.009266+0000","last_became_active":"2024-08-19T01:30:51.562211+0000","last_became_peered":"2024-08-19T01:30:51.562211+0000","last_unstale":"2024-08-19T01:33:14.009266+0000","last_undegraded":"2024-08-19T01:33:14.009266+0000","last_fullsized":"2024-08-19T01:33:14.009266+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:58.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929607+0000","last_change":"2024-08-19T01:30:49.575957+0000","last_active":"2024-08-19T01:33:13.929607+0000","last_peered":"2024-08-19T01:33:13.929607+0000","last_clean":"2024-08-19T01:33:13.929607+0000","last_became_active":"2024-08-19T01:30:49.575570+0000","last_became_peered":"2024-08-19T01:30:49.575570+0000","last_unstale":"2024-08-19T01:33:13.929607+0000","last_undegraded":"2024-08-19T01:33:13.929607+0000","last_fullsized":"2024-08-19T01:33:13.929607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:47.038113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821520+0000","last_change":"2024-08-19T01:32:35.826212+0000","last_active":"2024-08-19T01:33:13.821520+0000","last_peered":"2024-08-19T01:33:13.821520+0000","last_clean":"2024-08-19T01:33:13.821520+0000","last_became_active":"2024-08-19T01:32:35.825748+0000","last_became_peered":"2024-08-19T01:32:35.825748+0000","last_unstale":"2024-08-19T01:33:13.821520+0000","last_undegraded":"2024-08-19T01:33:13.821520+0000","last_fullsized":"2024-08-19T01:33:13.821520+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:34:59.072131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011949+0000","last_change":"2024-08-19T01:31:35.606449+0000","last_active":"2024-08-19T01:33:14.011949+0000","last_peered":"2024-08-19T01:33:14.011949+0000","last_clean":"2024-08-19T01:33:14.011949+0000","last_became_active":"2024-08-19T01:31:35.606105+0000","last_became_peered":"2024-08-19T01:31:35.606105+0000","last_unstale":"2024-08-19T01:33:14.011949+0000","last_undegraded":"2024-08-19T01:33:14.011949+0000","last_fullsized":"2024-08-19T01:33:14.011949+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:36:21.834763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010024+0000","last_change":"2024-08-19T01:30:49.542441+0000","last_active":"2024-08-19T01:33:14.010024+0000","last_peered":"2024-08-19T01:33:14.010024+0000","last_clean":"2024-08-19T01:33:14.010024+0000","last_became_active":"2024-08-19T01:30:49.542106+0000","last_became_peered":"2024-08-19T01:30:49.542106+0000","last_unstale":"2024-08-19T01:33:14.010024+0000","last_undegraded":"2024-08-19T01:33:14.010024+0000","last_fullsized":"2024-08-19T01:33:14.010024+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:21:01.956282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914501+0000","last_change":"2024-08-19T01:30:58.588899+0000","last_active":"2024-08-19T01:33:13.914501+0000","last_peered":"2024-08-19T01:33:13.914501+0000","last_clean":"2024-08-19T01:33:13.914501+0000","last_became_active":"2024-08-19T01:30:58.588521+0000","last_became_peered":"2024-08-19T01:30:58.588521+0000","last_unstale":"2024-08-19T01:33:13.914501+0000","last_undegraded":"2024-08-19T01:33:13.914501+0000","last_fullsized":"2024-08-19T01:33:13.914501+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:43:59.485456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009765+0000","last_change":"2024-08-19T01:30:51.557248+0000","last_active":"2024-08-19T01:33:14.009765+0000","last_peered":"2024-08-19T01:33:14.009765+0000","last_clean":"2024-08-19T01:33:14.009765+0000","last_became_active":"2024-08-19T01:30:51.556448+0000","last_became_peered":"2024-08-19T01:30:51.556448+0000","last_unstale":"2024-08-19T01:33:14.009765+0000","last_undegraded":"2024-08-19T01:33:14.009765+0000","last_fullsized":"2024-08-19T01:33:14.009765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:19:19.587955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":145,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932795+0000","last_change":"2024-08-19T01:30:49.540360+0000","last_active":"2024-08-19T01:33:13.932795+0000","last_peered":"2024-08-19T01:33:13.932795+0000","last_clean":"2024-08-19T01:33:13.932795+0000","last_became_active":"2024-08-19T01:30:49.539993+0000","last_became_peered":"2024-08-19T01:30:49.539993+0000","last_unstale":"2024-08-19T01:33:13.932795+0000","last_undegraded":"2024-08-19T01:33:13.932795+0000","last_fullsized":"2024-08-19T01:33:13.932795+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:05:28.308441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820478+0000","last_change":"2024-08-19T01:30:58.592387+0000","last_active":"2024-08-19T01:33:13.820478+0000","last_peered":"2024-08-19T01:33:13.820478+0000","last_clean":"2024-08-19T01:33:13.820478+0000","last_became_active":"2024-08-19T01:30:58.592031+0000","last_became_peered":"2024-08-19T01:30:58.592031+0000","last_unstale":"2024-08-19T01:33:13.820478+0000","last_undegraded":"2024-08-19T01:33:13.820478+0000","last_fullsized":"2024-08-19T01:33:13.820478+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:31:23.159012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011277+0000","last_change":"2024-08-19T01:30:58.595650+0000","last_active":"2024-08-19T01:33:14.011277+0000","last_peered":"2024-08-19T01:33:14.011277+0000","last_clean":"2024-08-19T01:33:14.011277+0000","last_became_active":"2024-08-19T01:30:58.595304+0000","last_became_peered":"2024-08-19T01:30:58.595304+0000","last_unstale":"2024-08-19T01:33:14.011277+0000","last_undegraded":"2024-08-19T01:33:14.011277+0000","last_fullsized":"2024-08-19T01:33:14.011277+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:23:53.517355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914930+0000","last_change":"2024-08-19T01:30:51.548683+0000","last_active":"2024-08-19T01:33:13.914930+0000","last_peered":"2024-08-19T01:33:13.914930+0000","last_clean":"2024-08-19T01:33:13.914930+0000","last_became_active":"2024-08-19T01:30:51.548382+0000","last_became_peered":"2024-08-19T01:30:51.548382+0000","last_unstale":"2024-08-19T01:33:13.914930+0000","last_undegraded":"2024-08-19T01:33:13.914930+0000","last_fullsized":"2024-08-19T01:33:13.914930+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:46:37.551017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929191+0000","last_change":"2024-08-19T01:30:49.540136+0000","last_active":"2024-08-19T01:33:13.929191+0000","last_peered":"2024-08-19T01:33:13.929191+0000","last_clean":"2024-08-19T01:33:13.929191+0000","last_became_active":"2024-08-19T01:30:49.539680+0000","last_became_peered":"2024-08-19T01:30:49.539680+0000","last_unstale":"2024-08-19T01:33:13.929191+0000","last_undegraded":"2024-08-19T01:33:13.929191+0000","last_fullsized":"2024-08-19T01:33:13.929191+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:25.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009451+0000","last_change":"2024-08-19T01:30:58.597130+0000","last_active":"2024-08-19T01:33:14.009451+0000","last_peered":"2024-08-19T01:33:14.009451+0000","last_clean":"2024-08-19T01:33:14.009451+0000","last_became_active":"2024-08-19T01:30:58.596834+0000","last_became_peered":"2024-08-19T01:30:58.596834+0000","last_unstale":"2024-08-19T01:33:14.009451+0000","last_undegraded":"2024-08-19T01:33:14.009451+0000","last_fullsized":"2024-08-19T01:33:14.009451+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:13:10.441421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820451+0000","last_change":"2024-08-19T01:30:51.555575+0000","last_active":"2024-08-19T01:33:13.820451+0000","last_peered":"2024-08-19T01:33:13.820451+0000","last_clean":"2024-08-19T01:33:13.820451+0000","last_became_active":"2024-08-19T01:30:51.555309+0000","last_became_peered":"2024-08-19T01:30:51.555309+0000","last_unstale":"2024-08-19T01:33:13.820451+0000","last_undegraded":"2024-08-19T01:33:13.820451+0000","last_fullsized":"2024-08-19T01:33:13.820451+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:11:25.515286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009452+0000","last_change":"2024-08-19T01:30:49.536844+0000","last_active":"2024-08-19T01:33:14.009452+0000","last_peered":"2024-08-19T01:33:14.009452+0000","last_clean":"2024-08-19T01:33:14.009452+0000","last_became_active":"2024-08-19T01:30:49.536496+0000","last_became_peered":"2024-08-19T01:30:49.536496+0000","last_unstale":"2024-08-19T01:33:14.009452+0000","last_undegraded":"2024-08-19T01:33:14.009452+0000","last_fullsized":"2024-08-19T01:33:14.009452+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:04.917188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932414+0000","last_change":"2024-08-19T01:30:49.539356+0000","last_active":"2024-08-19T01:33:13.932414+0000","last_peered":"2024-08-19T01:33:13.932414+0000","last_clean":"2024-08-19T01:33:13.932414+0000","last_became_active":"2024-08-19T01:30:49.538953+0000","last_became_peered":"2024-08-19T01:30:49.538953+0000","last_unstale":"2024-08-19T01:33:13.932414+0000","last_undegraded":"2024-08-19T01:33:13.932414+0000","last_fullsized":"2024-08-19T01:33:13.932414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:13:17.845140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929202+0000","last_change":"2024-08-19T01:31:35.870021+0000","last_active":"2024-08-19T01:33:13.929202+0000","last_peered":"2024-08-19T01:33:13.929202+0000","last_clean":"2024-08-19T01:33:13.929202+0000","last_became_active":"2024-08-19T01:31:35.869742+0000","last_became_peered":"2024-08-19T01:31:35.869742+0000","last_unstale":"2024-08-19T01:33:13.929202+0000","last_undegraded":"2024-08-19T01:33:13.929202+0000","last_fullsized":"2024-08-19T01:33:13.929202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:51.848694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821878+0000","last_change":"2024-08-19T01:30:58.590385+0000","last_active":"2024-08-19T01:33:13.821878+0000","last_peered":"2024-08-19T01:33:13.821878+0000","last_clean":"2024-08-19T01:33:13.821878+0000","last_became_active":"2024-08-19T01:30:58.590004+0000","last_became_peered":"2024-08-19T01:30:58.590004+0000","last_unstale":"2024-08-19T01:33:13.821878+0000","last_undegraded":"2024-08-19T01:33:13.821878+0000","last_fullsized":"2024-08-19T01:33:13.821878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:13:01.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.739173+0000","last_change":"2024-08-19T01:31:35.614962+0000","last_active":"2024-08-19T01:33:14.739173+0000","last_peered":"2024-08-19T01:33:14.739173+0000","last_clean":"2024-08-19T01:33:14.739173+0000","last_became_active":"2024-08-19T01:31:35.605904+0000","last_became_peered":"2024-08-19T01:31:35.605904+0000","last_unstale":"2024-08-19T01:33:14.739173+0000","last_undegraded":"2024-08-19T01:33:14.739173+0000","last_fullsized":"2024-08-19T01:33:14.739173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:26:36.102805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":3,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920351+0000","last_change":"2024-08-19T01:30:51.550448+0000","last_active":"2024-08-19T01:33:13.920351+0000","last_peered":"2024-08-19T01:33:13.920351+0000","last_clean":"2024-08-19T01:33:13.920351+0000","last_became_active":"2024-08-19T01:30:51.550074+0000","last_became_peered":"2024-08-19T01:30:51.550074+0000","last_unstale":"2024-08-19T01:33:13.920351+0000","last_undegraded":"2024-08-19T01:33:13.920351+0000","last_fullsized":"2024-08-19T01:33:13.920351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:33:13.016183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822063+0000","last_change":"2024-08-19T01:30:58.592077+0000","last_active":"2024-08-19T01:33:13.822063+0000","last_peered":"2024-08-19T01:33:13.822063+0000","last_clean":"2024-08-19T01:33:13.822063+0000","last_became_active":"2024-08-19T01:30:58.591651+0000","last_became_peered":"2024-08-19T01:30:58.591651+0000","last_unstale":"2024-08-19T01:33:13.822063+0000","last_undegraded":"2024-08-19T01:33:13.822063+0000","last_fullsized":"2024-08-19T01:33:13.822063+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:09:24.980954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009090+0000","last_change":"2024-08-19T01:30:51.552772+0000","last_active":"2024-08-19T01:33:14.009090+0000","last_peered":"2024-08-19T01:33:14.009090+0000","last_clean":"2024-08-19T01:33:14.009090+0000","last_became_active":"2024-08-19T01:30:51.550904+0000","last_became_peered":"2024-08-19T01:30:51.550904+0000","last_unstale":"2024-08-19T01:33:14.009090+0000","last_undegraded":"2024-08-19T01:33:14.009090+0000","last_fullsized":"2024-08-19T01:33:14.009090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:21:50.082587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821870+0000","last_change":"2024-08-19T01:30:51.559756+0000","last_active":"2024-08-19T01:33:13.821870+0000","last_peered":"2024-08-19T01:33:13.821870+0000","last_clean":"2024-08-19T01:33:13.821870+0000","last_became_active":"2024-08-19T01:30:51.559370+0000","last_became_peered":"2024-08-19T01:30:51.559370+0000","last_unstale":"2024-08-19T01:33:13.821870+0000","last_undegraded":"2024-08-19T01:33:13.821870+0000","last_fullsized":"2024-08-19T01:33:13.821870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:27:30.525346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821589+0000","last_change":"2024-08-19T01:30:51.556577+0000","last_active":"2024-08-19T01:33:13.821589+0000","last_peered":"2024-08-19T01:33:13.821589+0000","last_clean":"2024-08-19T01:33:13.821589+0000","last_became_active":"2024-08-19T01:30:51.555929+0000","last_became_peered":"2024-08-19T01:30:51.555929+0000","last_unstale":"2024-08-19T01:33:13.821589+0000","last_undegraded":"2024-08-19T01:33:13.821589+0000","last_fullsized":"2024-08-19T01:33:13.821589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:46:31.660752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010627+0000","last_change":"2024-08-19T01:30:51.558626+0000","last_active":"2024-08-19T01:33:14.010627+0000","last_peered":"2024-08-19T01:33:14.010627+0000","last_clean":"2024-08-19T01:33:14.010627+0000","last_became_active":"2024-08-19T01:30:51.558188+0000","last_became_peered":"2024-08-19T01:30:51.558188+0000","last_unstale":"2024-08-19T01:33:14.010627+0000","last_undegraded":"2024-08-19T01:33:14.010627+0000","last_fullsized":"2024-08-19T01:33:14.010627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:43:44.777643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010095+0000","last_change":"2024-08-19T01:30:51.552870+0000","last_active":"2024-08-19T01:33:14.010095+0000","last_peered":"2024-08-19T01:33:14.010095+0000","last_clean":"2024-08-19T01:33:14.010095+0000","last_became_active":"2024-08-19T01:30:51.552514+0000","last_became_peered":"2024-08-19T01:30:51.552514+0000","last_unstale":"2024-08-19T01:33:14.010095+0000","last_undegraded":"2024-08-19T01:33:14.010095+0000","last_fullsized":"2024-08-19T01:33:14.010095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:10:58.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009491+0000","last_change":"2024-08-19T01:30:51.554216+0000","last_active":"2024-08-19T01:33:14.009491+0000","last_peered":"2024-08-19T01:33:14.009491+0000","last_clean":"2024-08-19T01:33:14.009491+0000","last_became_active":"2024-08-19T01:30:51.552535+0000","last_became_peered":"2024-08-19T01:30:51.552535+0000","last_unstale":"2024-08-19T01:33:14.009491+0000","last_undegraded":"2024-08-19T01:33:14.009491+0000","last_fullsized":"2024-08-19T01:33:14.009491+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:26:52.971897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822254+0000","last_change":"2024-08-19T01:30:51.561703+0000","last_active":"2024-08-19T01:33:13.822254+0000","last_peered":"2024-08-19T01:33:13.822254+0000","last_clean":"2024-08-19T01:33:13.822254+0000","last_became_active":"2024-08-19T01:30:51.561385+0000","last_became_peered":"2024-08-19T01:30:51.561385+0000","last_unstale":"2024-08-19T01:33:13.822254+0000","last_undegraded":"2024-08-19T01:33:13.822254+0000","last_fullsized":"2024-08-19T01:33:13.822254+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:07:23.187023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820799+0000","last_change":"2024-08-19T01:30:51.564047+0000","last_active":"2024-08-19T01:33:13.820799+0000","last_peered":"2024-08-19T01:33:13.820799+0000","last_clean":"2024-08-19T01:33:13.820799+0000","last_became_active":"2024-08-19T01:30:51.563743+0000","last_became_peered":"2024-08-19T01:30:51.563743+0000","last_unstale":"2024-08-19T01:33:13.820799+0000","last_undegraded":"2024-08-19T01:33:13.820799+0000","last_fullsized":"2024-08-19T01:33:13.820799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:50:37.639496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929073+0000","last_change":"2024-08-19T01:30:51.557058+0000","last_active":"2024-08-19T01:33:13.929073+0000","last_peered":"2024-08-19T01:33:13.929073+0000","last_clean":"2024-08-19T01:33:13.929073+0000","last_became_active":"2024-08-19T01:30:51.554611+0000","last_became_peered":"2024-08-19T01:30:51.554611+0000","last_unstale":"2024-08-19T01:33:13.929073+0000","last_undegraded":"2024-08-19T01:33:13.929073+0000","last_fullsized":"2024-08-19T01:33:13.929073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:56:25.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820511+0000","last_change":"2024-08-19T01:30:51.564119+0000","last_active":"2024-08-19T01:33:13.820511+0000","last_peered":"2024-08-19T01:33:13.820511+0000","last_clean":"2024-08-19T01:33:13.820511+0000","last_became_active":"2024-08-19T01:30:51.563840+0000","last_became_peered":"2024-08-19T01:30:51.563840+0000","last_unstale":"2024-08-19T01:33:13.820511+0000","last_undegraded":"2024-08-19T01:33:13.820511+0000","last_fullsized":"2024-08-19T01:33:13.820511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:44:44.928760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009253+0000","last_change":"2024-08-19T01:30:51.561639+0000","last_active":"2024-08-19T01:33:14.009253+0000","last_peered":"2024-08-19T01:33:14.009253+0000","last_clean":"2024-08-19T01:33:14.009253+0000","last_became_active":"2024-08-19T01:30:51.561207+0000","last_became_peered":"2024-08-19T01:30:51.561207+0000","last_unstale":"2024-08-19T01:33:14.009253+0000","last_undegraded":"2024-08-19T01:33:14.009253+0000","last_fullsized":"2024-08-19T01:33:14.009253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:12:20.076628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011400+0000","last_change":"2024-08-19T01:30:51.556700+0000","last_active":"2024-08-19T01:33:14.011400+0000","last_peered":"2024-08-19T01:33:14.011400+0000","last_clean":"2024-08-19T01:33:14.011400+0000","last_became_active":"2024-08-19T01:30:51.556407+0000","last_became_peered":"2024-08-19T01:30:51.556407+0000","last_unstale":"2024-08-19T01:33:14.011400+0000","last_undegraded":"2024-08-19T01:33:14.011400+0000","last_fullsized":"2024-08-19T01:33:14.011400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:03:16.664253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009274+0000","last_change":"2024-08-19T01:30:51.553764+0000","last_active":"2024-08-19T01:33:14.009274+0000","last_peered":"2024-08-19T01:33:14.009274+0000","last_clean":"2024-08-19T01:33:14.009274+0000","last_became_active":"2024-08-19T01:30:51.553442+0000","last_became_peered":"2024-08-19T01:30:51.553442+0000","last_unstale":"2024-08-19T01:33:14.009274+0000","last_undegraded":"2024-08-19T01:33:14.009274+0000","last_fullsized":"2024-08-19T01:33:14.009274+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:36:44.476979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822255+0000","last_change":"2024-08-19T01:30:51.565528+0000","last_active":"2024-08-19T01:33:13.822255+0000","last_peered":"2024-08-19T01:33:13.822255+0000","last_clean":"2024-08-19T01:33:13.822255+0000","last_became_active":"2024-08-19T01:30:51.564649+0000","last_became_peered":"2024-08-19T01:30:51.564649+0000","last_unstale":"2024-08-19T01:33:13.822255+0000","last_undegraded":"2024-08-19T01:33:13.822255+0000","last_fullsized":"2024-08-19T01:33:13.822255+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:04:12.868496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.917383+0000","last_change":"2024-08-19T01:30:51.553425+0000","last_active":"2024-08-19T01:33:13.917383+0000","last_peered":"2024-08-19T01:33:13.917383+0000","last_clean":"2024-08-19T01:33:13.917383+0000","last_became_active":"2024-08-19T01:30:51.553123+0000","last_became_peered":"2024-08-19T01:30:51.553123+0000","last_unstale":"2024-08-19T01:33:13.917383+0000","last_undegraded":"2024-08-19T01:33:13.917383+0000","last_fullsized":"2024-08-19T01:33:13.917383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:07.969228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009636+0000","last_change":"2024-08-19T01:30:51.556820+0000","last_active":"2024-08-19T01:33:14.009636+0000","last_peered":"2024-08-19T01:33:14.009636+0000","last_clean":"2024-08-19T01:33:14.009636+0000","last_became_active":"2024-08-19T01:30:51.556532+0000","last_became_peered":"2024-08-19T01:30:51.556532+0000","last_unstale":"2024-08-19T01:33:14.009636+0000","last_undegraded":"2024-08-19T01:33:14.009636+0000","last_fullsized":"2024-08-19T01:33:14.009636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:40:37.053193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.008986+0000","last_change":"2024-08-19T01:30:51.561610+0000","last_active":"2024-08-19T01:33:14.008986+0000","last_peered":"2024-08-19T01:33:14.008986+0000","last_clean":"2024-08-19T01:33:14.008986+0000","last_became_active":"2024-08-19T01:30:51.561270+0000","last_became_peered":"2024-08-19T01:30:51.561270+0000","last_unstale":"2024-08-19T01:33:14.008986+0000","last_undegraded":"2024-08-19T01:33:14.008986+0000","last_fullsized":"2024-08-19T01:33:14.008986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:15:15.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010003+0000","last_change":"2024-08-19T01:30:51.558128+0000","last_active":"2024-08-19T01:33:14.010003+0000","last_peered":"2024-08-19T01:33:14.010003+0000","last_clean":"2024-08-19T01:33:14.010003+0000","last_became_active":"2024-08-19T01:30:51.557785+0000","last_became_peered":"2024-08-19T01:30:51.557785+0000","last_unstale":"2024-08-19T01:33:14.010003+0000","last_undegraded":"2024-08-19T01:33:14.010003+0000","last_fullsized":"2024-08-19T01:33:14.010003+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:14.849212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010225+0000","last_change":"2024-08-19T01:30:51.553917+0000","last_active":"2024-08-19T01:33:14.010225+0000","last_peered":"2024-08-19T01:33:14.010225+0000","last_clean":"2024-08-19T01:33:14.010225+0000","last_became_active":"2024-08-19T01:30:51.553596+0000","last_became_peered":"2024-08-19T01:30:51.553596+0000","last_unstale":"2024-08-19T01:33:14.010225+0000","last_undegraded":"2024-08-19T01:33:14.010225+0000","last_fullsized":"2024-08-19T01:33:14.010225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:03:36.001486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011300+0000","last_change":"2024-08-19T01:30:51.555201+0000","last_active":"2024-08-19T01:33:14.011300+0000","last_peered":"2024-08-19T01:33:14.011300+0000","last_clean":"2024-08-19T01:33:14.011300+0000","last_became_active":"2024-08-19T01:30:51.554160+0000","last_became_peered":"2024-08-19T01:30:51.554160+0000","last_unstale":"2024-08-19T01:33:14.011300+0000","last_undegraded":"2024-08-19T01:33:14.011300+0000","last_fullsized":"2024-08-19T01:33:14.011300+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:58:20.586931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821883+0000","last_change":"2024-08-19T01:30:51.560517+0000","last_active":"2024-08-19T01:33:13.821883+0000","last_peered":"2024-08-19T01:33:13.821883+0000","last_clean":"2024-08-19T01:33:13.821883+0000","last_became_active":"2024-08-19T01:30:51.557454+0000","last_became_peered":"2024-08-19T01:30:51.557454+0000","last_unstale":"2024-08-19T01:33:13.821883+0000","last_undegraded":"2024-08-19T01:33:13.821883+0000","last_fullsized":"2024-08-19T01:33:13.821883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:41:08.192165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010127+0000","last_change":"2024-08-19T01:30:51.565961+0000","last_active":"2024-08-19T01:33:14.010127+0000","last_peered":"2024-08-19T01:33:14.010127+0000","last_clean":"2024-08-19T01:33:14.010127+0000","last_became_active":"2024-08-19T01:30:51.565491+0000","last_became_peered":"2024-08-19T01:30:51.565491+0000","last_unstale":"2024-08-19T01:33:14.010127+0000","last_undegraded":"2024-08-19T01:33:14.010127+0000","last_fullsized":"2024-08-19T01:33:14.010127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:53:06.263411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932774+0000","last_change":"2024-08-19T01:30:51.551297+0000","last_active":"2024-08-19T01:33:13.932774+0000","last_peered":"2024-08-19T01:33:13.932774+0000","last_clean":"2024-08-19T01:33:13.932774+0000","last_became_active":"2024-08-19T01:30:51.550929+0000","last_became_peered":"2024-08-19T01:30:51.550929+0000","last_unstale":"2024-08-19T01:33:13.932774+0000","last_undegraded":"2024-08-19T01:33:13.932774+0000","last_fullsized":"2024-08-19T01:33:13.932774+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:07:03.625360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916372+0000","last_change":"2024-08-19T01:31:35.602883+0000","last_active":"2024-08-19T01:33:13.916372+0000","last_peered":"2024-08-19T01:33:13.916372+0000","last_clean":"2024-08-19T01:33:13.916372+0000","last_became_active":"2024-08-19T01:31:35.602571+0000","last_became_peered":"2024-08-19T01:31:35.602571+0000","last_unstale":"2024-08-19T01:33:13.916372+0000","last_undegraded":"2024-08-19T01:33:13.916372+0000","last_fullsized":"2024-08-19T01:33:13.916372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:08:13.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821015+0000","last_change":"2024-08-19T01:30:51.552548+0000","last_active":"2024-08-19T01:33:13.821015+0000","last_peered":"2024-08-19T01:33:13.821015+0000","last_clean":"2024-08-19T01:33:13.821015+0000","last_became_active":"2024-08-19T01:30:51.552264+0000","last_became_peered":"2024-08-19T01:30:51.552264+0000","last_unstale":"2024-08-19T01:33:13.821015+0000","last_undegraded":"2024-08-19T01:33:13.821015+0000","last_fullsized":"2024-08-19T01:33:13.821015+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:17:53.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009360+0000","last_change":"2024-08-19T01:30:51.558686+0000","last_active":"2024-08-19T01:33:14.009360+0000","last_peered":"2024-08-19T01:33:14.009360+0000","last_clean":"2024-08-19T01:33:14.009360+0000","last_became_active":"2024-08-19T01:30:51.558190+0000","last_became_peered":"2024-08-19T01:30:51.558190+0000","last_unstale":"2024-08-19T01:33:14.009360+0000","last_undegraded":"2024-08-19T01:33:14.009360+0000","last_fullsized":"2024-08-19T01:33:14.009360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:52:33.056948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932373+0000","last_change":"2024-08-19T01:30:51.548868+0000","last_active":"2024-08-19T01:33:13.932373+0000","last_peered":"2024-08-19T01:33:13.932373+0000","last_clean":"2024-08-19T01:33:13.932373+0000","last_became_active":"2024-08-19T01:30:51.548347+0000","last_became_peered":"2024-08-19T01:30:51.548347+0000","last_unstale":"2024-08-19T01:33:13.932373+0000","last_undegraded":"2024-08-19T01:33:13.932373+0000","last_fullsized":"2024-08-19T01:33:13.932373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:39:22.049290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920367+0000","last_change":"2024-08-19T01:30:51.554054+0000","last_active":"2024-08-19T01:33:13.920367+0000","last_peered":"2024-08-19T01:33:13.920367+0000","last_clean":"2024-08-19T01:33:13.920367+0000","last_became_active":"2024-08-19T01:30:51.553731+0000","last_became_peered":"2024-08-19T01:30:51.553731+0000","last_unstale":"2024-08-19T01:33:13.920367+0000","last_undegraded":"2024-08-19T01:33:13.920367+0000","last_fullsized":"2024-08-19T01:33:13.920367+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:25:17.939024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821646+0000","last_change":"2024-08-19T01:30:51.558662+0000","last_active":"2024-08-19T01:33:13.821646+0000","last_peered":"2024-08-19T01:33:13.821646+0000","last_clean":"2024-08-19T01:33:13.821646+0000","last_became_active":"2024-08-19T01:30:51.558131+0000","last_became_peered":"2024-08-19T01:30:51.558131+0000","last_unstale":"2024-08-19T01:33:13.821646+0000","last_undegraded":"2024-08-19T01:33:13.821646+0000","last_fullsized":"2024-08-19T01:33:13.821646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:31:25.054005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011016+0000","last_change":"2024-08-19T01:30:51.565170+0000","last_active":"2024-08-19T01:33:14.011016+0000","last_peered":"2024-08-19T01:33:14.011016+0000","last_clean":"2024-08-19T01:33:14.011016+0000","last_became_active":"2024-08-19T01:30:51.564765+0000","last_became_peered":"2024-08-19T01:30:51.564765+0000","last_unstale":"2024-08-19T01:33:14.011016+0000","last_undegraded":"2024-08-19T01:33:14.011016+0000","last_fullsized":"2024-08-19T01:33:14.011016+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:00:48.668865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011370+0000","last_change":"2024-08-19T01:30:51.555300+0000","last_active":"2024-08-19T01:33:14.011370+0000","last_peered":"2024-08-19T01:33:14.011370+0000","last_clean":"2024-08-19T01:33:14.011370+0000","last_became_active":"2024-08-19T01:30:51.554977+0000","last_became_peered":"2024-08-19T01:30:51.554977+0000","last_unstale":"2024-08-19T01:33:14.011370+0000","last_undegraded":"2024-08-19T01:33:14.011370+0000","last_fullsized":"2024-08-19T01:33:14.011370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:18:58.961783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009590+0000","last_change":"2024-08-19T01:30:51.557906+0000","last_active":"2024-08-19T01:33:14.009590+0000","last_peered":"2024-08-19T01:33:14.009590+0000","last_clean":"2024-08-19T01:33:14.009590+0000","last_became_active":"2024-08-19T01:30:51.557598+0000","last_became_peered":"2024-08-19T01:30:51.557598+0000","last_unstale":"2024-08-19T01:33:14.009590+0000","last_undegraded":"2024-08-19T01:33:14.009590+0000","last_fullsized":"2024-08-19T01:33:14.009590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:33:12.785910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599828,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":727,"num_read_kb":472,"num_write":298,"num_write_kb":1445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4463,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1799484,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107877,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481833,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":542338,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743470,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32364,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712788,"statfs":{"total":95995035648,"available":95961894912,"internally_reserved":0,"allocated":1216512,"data_stored":1335393,"data_compressed":74600,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906995,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":935790,"data_compressed":51631,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168630,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":935710,"data_compressed":51631,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430268,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":1140634,"data_compressed":68444,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593793,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":537750,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822726,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1266511,"data_compressed":74600,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712872,"statfs":{"total":95995035648,"available":95961980928,"internally_reserved":0,"allocated":1126400,"data_stored":612491,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215181,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":537944,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":528464,"data_compressed":29513,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":608373,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":529214,"data_compressed":29513,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":70713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134387,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":203269,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T01:33:19.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T01:33:19.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:18 smithi161 ceph-mon[21757]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:33:19.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:18 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:33:19.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:18 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:33:19.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:18 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:33:19.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:18 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:33:19.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:18 smithi161 ceph-mon[21757]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:19.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:18 smithi161 ceph-mon[21757]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-08-19T01:33:19.459 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 -- 172.21.15.64:0/3181949893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 msgr2=0x7f5edc101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 --2- 172.21.15.64:0/3181949893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc101d10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009930 tx=0x7f5ecc02a420 comp rx=0 tx=0).stop 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 -- 172.21.15.64:0/3181949893 shutdown_connections 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 --2- 172.21.15.64:0/3181949893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5edc102250 0x7f5edc10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 --2- 172.21.15.64:0/3181949893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc101d10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 --2- 172.21.15.64:0/3181949893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc1078b0 0x7f5edc107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 -- 172.21.15.64:0/3181949893 >> 172.21.15.64:0/3181949893 conn(0x7f5edc0fd5f0 msgr2=0x7f5edc0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:19.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 -- 172.21.15.64:0/3181949893 shutdown_connections 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.942+0000 7f5ee447e640 1 -- 172.21.15.64:0/3181949893 wait complete. 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 Processor -- start 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 -- start start 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc102250 0x7f5edc074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5edc1078b0 0x7f5edc074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc111d50 con 0x7f5edc1018b0 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc111bd0 con 0x7f5edc102250 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee447e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc111ed0 con 0x7f5edc1078b0 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee221a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee221a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc074220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50872/0 (socket says 172.21.15.64:50872) 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee221a640 1 -- 172.21.15.64:0/1663519587 learned_addr learned my addr 172.21.15.64:0/1663519587 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee2a1b640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5edc1078b0 0x7f5edc074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:19.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.943+0000 7f5ee1a19640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc102250 0x7f5edc074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee221a640 1 -- 172.21.15.64:0/1663519587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5edc1078b0 msgr2=0x7f5edc074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee221a640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5edc1078b0 0x7f5edc074ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee221a640 1 -- 172.21.15.64:0/1663519587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc102250 msgr2=0x7f5edc074760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee221a640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc102250 0x7f5edc074760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee221a640 1 -- 172.21.15.64:0/1663519587 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc1a6810 con 0x7f5edc1018b0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee2a1b640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5edc1078b0 0x7f5edc074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee221a640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc074220 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800b4a0 tx=0x7f5ed800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ed37fe640 1 -- 172.21.15.64:0/1663519587 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8014070 con 0x7f5edc1018b0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc1a6b00 con 0x7f5edc1018b0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ed37fe640 1 -- 172.21.15.64:0/1663519587 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ed8010af0 con 0x7f5edc1018b0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ed37fe640 1 -- 172.21.15.64:0/1663519587 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8017d10 con 0x7f5edc1018b0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.944+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc1a7090 con 0x7f5edc1018b0 2024-08-19T01:33:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.945+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc101d90 con 0x7f5edc1018b0 2024-08-19T01:33:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.946+0000 7f5ed37fe640 1 -- 172.21.15.64:0/1663519587 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f5ed8017070 con 0x7f5edc1018b0 2024-08-19T01:33:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.947+0000 7f5ed37fe640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5ec80a2e10 0x7f5ec80a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.947+0000 7f5ee1a19640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5ec80a2e10 0x7f5ec80a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.947+0000 7f5ed37fe640 1 -- 172.21.15.64:0/1663519587 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5ed80cedc0 con 0x7f5edc1018b0 2024-08-19T01:33:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.947+0000 7f5ee1a19640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5ec80a2e10 0x7f5ec80a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0096c0 tx=0x7f5ecc000bc0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:19.951+0000 7f5ed37fe640 1 -- 172.21.15.64:0/1663519587 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f5ed809b4f0 con 0x7f5edc1018b0 2024-08-19T01:33:20.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.122+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5edc062360 con 0x7f5ec80a2e10 2024-08-19T01:33:20.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.131+0000 7f5ed37fe640 1 -- 172.21.15.64:0/1663519587 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339676 (secure 0 0 0) 0x7f5edc062360 con 0x7f5ec80a2e10 2024-08-19T01:33:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:20.138 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-08-19T01:33:20.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5ec80a2e10 msgr2=0x7f5ec80a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5ec80a2e10 0x7f5ec80a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0096c0 tx=0x7f5ecc000bc0 comp rx=0 tx=0).stop 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 msgr2=0x7f5edc074220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc074220 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800b4a0 tx=0x7f5ed800b960 comp rx=0 tx=0).stop 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 shutdown_connections 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5ec80a2e10 0x7f5ec80a52d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5edc1078b0 0x7f5edc074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc102250 0x7f5edc074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 --2- 172.21.15.64:0/1663519587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5edc1018b0 0x7f5edc074220 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.134+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 >> 172.21.15.64:0/1663519587 conn(0x7f5edc0fd5f0 msgr2=0x7f5edc0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.135+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 shutdown_connections 2024-08-19T01:33:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:20.135+0000 7f5ee447e640 1 -- 172.21.15.64:0/1663519587 wait complete. 2024-08-19T01:33:20.890 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-08-19T01:33:19.833200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190430,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":868,"num_read_kb":596,"num_write":481,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1185316,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361952,"kb_used_data":12768,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579872,"statfs":{"total":1151940427776,"available":1151569788928,"internally_reserved":0,"allocated":13074432,"data_stored":9510612,"data_compressed":542357,"data_compressed_allocated":733184,"data_compressed_original":3866624,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.991561"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916960+0000","last_change":"2024-08-19T01:30:49.539512+0000","last_active":"2024-08-19T01:33:13.916960+0000","last_peered":"2024-08-19T01:33:13.916960+0000","last_clean":"2024-08-19T01:33:13.916960+0000","last_became_active":"2024-08-19T01:30:49.539163+0000","last_became_peered":"2024-08-19T01:30:49.539163+0000","last_unstale":"2024-08-19T01:33:13.916960+0000","last_undegraded":"2024-08-19T01:33:13.916960+0000","last_fullsized":"2024-08-19T01:33:13.916960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:23.874916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821250+0000","last_change":"2024-08-19T01:30:51.561135+0000","last_active":"2024-08-19T01:33:13.821250+0000","last_peered":"2024-08-19T01:33:13.821250+0000","last_clean":"2024-08-19T01:33:13.821250+0000","last_became_active":"2024-08-19T01:30:51.560822+0000","last_became_peered":"2024-08-19T01:30:51.560822+0000","last_unstale":"2024-08-19T01:33:13.821250+0000","last_undegraded":"2024-08-19T01:33:13.821250+0000","last_fullsized":"2024-08-19T01:33:13.821250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:08:35.918895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821530+0000","last_change":"2024-08-19T01:30:58.595727+0000","last_active":"2024-08-19T01:33:13.821530+0000","last_peered":"2024-08-19T01:33:13.821530+0000","last_clean":"2024-08-19T01:33:13.821530+0000","last_became_active":"2024-08-19T01:30:58.595286+0000","last_became_peered":"2024-08-19T01:30:58.595286+0000","last_unstale":"2024-08-19T01:33:13.821530+0000","last_undegraded":"2024-08-19T01:33:13.821530+0000","last_fullsized":"2024-08-19T01:33:13.821530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:25:02.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821435+0000","last_change":"2024-08-19T01:30:49.575693+0000","last_active":"2024-08-19T01:33:13.821435+0000","last_peered":"2024-08-19T01:33:13.821435+0000","last_clean":"2024-08-19T01:33:13.821435+0000","last_became_active":"2024-08-19T01:30:49.575331+0000","last_became_peered":"2024-08-19T01:30:49.575331+0000","last_unstale":"2024-08-19T01:33:13.821435+0000","last_undegraded":"2024-08-19T01:33:13.821435+0000","last_fullsized":"2024-08-19T01:33:13.821435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:31:50.773160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011817+0000","last_change":"2024-08-19T01:30:51.562446+0000","last_active":"2024-08-19T01:33:14.011817+0000","last_peered":"2024-08-19T01:33:14.011817+0000","last_clean":"2024-08-19T01:33:14.011817+0000","last_became_active":"2024-08-19T01:30:51.561475+0000","last_became_peered":"2024-08-19T01:30:51.561475+0000","last_unstale":"2024-08-19T01:33:14.011817+0000","last_undegraded":"2024-08-19T01:33:14.011817+0000","last_fullsized":"2024-08-19T01:33:14.011817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:51:41.531247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821367+0000","last_change":"2024-08-19T01:30:58.591371+0000","last_active":"2024-08-19T01:33:13.821367+0000","last_peered":"2024-08-19T01:33:13.821367+0000","last_clean":"2024-08-19T01:33:13.821367+0000","last_became_active":"2024-08-19T01:30:58.590936+0000","last_became_peered":"2024-08-19T01:30:58.590936+0000","last_unstale":"2024-08-19T01:33:13.821367+0000","last_undegraded":"2024-08-19T01:33:13.821367+0000","last_fullsized":"2024-08-19T01:33:13.821367+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:07:33.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009287+0000","last_change":"2024-08-19T01:31:35.613145+0000","last_active":"2024-08-19T01:33:14.009287+0000","last_peered":"2024-08-19T01:33:14.009287+0000","last_clean":"2024-08-19T01:33:14.009287+0000","last_became_active":"2024-08-19T01:31:35.605087+0000","last_became_peered":"2024-08-19T01:31:35.605087+0000","last_unstale":"2024-08-19T01:33:14.009287+0000","last_undegraded":"2024-08-19T01:33:14.009287+0000","last_fullsized":"2024-08-19T01:33:14.009287+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:25:59.109707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009849+0000","last_change":"2024-08-19T01:30:51.555256+0000","last_active":"2024-08-19T01:33:14.009849+0000","last_peered":"2024-08-19T01:33:14.009849+0000","last_clean":"2024-08-19T01:33:14.009849+0000","last_became_active":"2024-08-19T01:30:51.554030+0000","last_became_peered":"2024-08-19T01:30:51.554030+0000","last_unstale":"2024-08-19T01:33:14.009849+0000","last_undegraded":"2024-08-19T01:33:14.009849+0000","last_fullsized":"2024-08-19T01:33:14.009849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:12:04.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821647+0000","last_change":"2024-08-19T01:30:58.589362+0000","last_active":"2024-08-19T01:33:13.821647+0000","last_peered":"2024-08-19T01:33:13.821647+0000","last_clean":"2024-08-19T01:33:13.821647+0000","last_became_active":"2024-08-19T01:30:58.588937+0000","last_became_peered":"2024-08-19T01:30:58.588937+0000","last_unstale":"2024-08-19T01:33:13.821647+0000","last_undegraded":"2024-08-19T01:33:13.821647+0000","last_fullsized":"2024-08-19T01:33:13.821647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:31.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920588+0000","last_change":"2024-08-19T01:30:49.538597+0000","last_active":"2024-08-19T01:33:13.920588+0000","last_peered":"2024-08-19T01:33:13.920588+0000","last_clean":"2024-08-19T01:33:13.920588+0000","last_became_active":"2024-08-19T01:30:49.538228+0000","last_became_peered":"2024-08-19T01:30:49.538228+0000","last_unstale":"2024-08-19T01:33:13.920588+0000","last_undegraded":"2024-08-19T01:33:13.920588+0000","last_fullsized":"2024-08-19T01:33:13.920588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:31.350962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929475+0000","last_change":"2024-08-19T01:30:51.562829+0000","last_active":"2024-08-19T01:33:13.929475+0000","last_peered":"2024-08-19T01:33:13.929475+0000","last_clean":"2024-08-19T01:33:13.929475+0000","last_became_active":"2024-08-19T01:30:51.562144+0000","last_became_peered":"2024-08-19T01:30:51.562144+0000","last_unstale":"2024-08-19T01:33:13.929475+0000","last_undegraded":"2024-08-19T01:33:13.929475+0000","last_fullsized":"2024-08-19T01:33:13.929475+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:17:09.184222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821477+0000","last_change":"2024-08-19T01:30:58.593178+0000","last_active":"2024-08-19T01:33:13.821477+0000","last_peered":"2024-08-19T01:33:13.821477+0000","last_clean":"2024-08-19T01:33:13.821477+0000","last_became_active":"2024-08-19T01:30:58.591247+0000","last_became_peered":"2024-08-19T01:30:58.591247+0000","last_unstale":"2024-08-19T01:33:13.821477+0000","last_undegraded":"2024-08-19T01:33:13.821477+0000","last_fullsized":"2024-08-19T01:33:13.821477+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:20:40.796135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009302+0000","last_change":"2024-08-19T01:30:51.562307+0000","last_active":"2024-08-19T01:33:14.009302+0000","last_peered":"2024-08-19T01:33:14.009302+0000","last_clean":"2024-08-19T01:33:14.009302+0000","last_became_active":"2024-08-19T01:30:51.561957+0000","last_became_peered":"2024-08-19T01:30:51.561957+0000","last_unstale":"2024-08-19T01:33:14.009302+0000","last_undegraded":"2024-08-19T01:33:14.009302+0000","last_fullsized":"2024-08-19T01:33:14.009302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:22.902317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009176+0000","last_change":"2024-08-19T01:31:35.609612+0000","last_active":"2024-08-19T01:33:14.009176+0000","last_peered":"2024-08-19T01:33:14.009176+0000","last_clean":"2024-08-19T01:33:14.009176+0000","last_became_active":"2024-08-19T01:31:35.604969+0000","last_became_peered":"2024-08-19T01:31:35.604969+0000","last_unstale":"2024-08-19T01:33:14.009176+0000","last_undegraded":"2024-08-19T01:33:14.009176+0000","last_fullsized":"2024-08-19T01:33:14.009176+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:59:38.089119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821743+0000","last_change":"2024-08-19T01:30:58.588984+0000","last_active":"2024-08-19T01:33:13.821743+0000","last_peered":"2024-08-19T01:33:13.821743+0000","last_clean":"2024-08-19T01:33:13.821743+0000","last_became_active":"2024-08-19T01:30:58.588504+0000","last_became_peered":"2024-08-19T01:30:58.588504+0000","last_unstale":"2024-08-19T01:33:13.821743+0000","last_undegraded":"2024-08-19T01:33:13.821743+0000","last_fullsized":"2024-08-19T01:33:13.821743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:29:14.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820715+0000","last_change":"2024-08-19T01:30:51.553293+0000","last_active":"2024-08-19T01:33:13.820715+0000","last_peered":"2024-08-19T01:33:13.820715+0000","last_clean":"2024-08-19T01:33:13.820715+0000","last_became_active":"2024-08-19T01:30:51.552990+0000","last_became_peered":"2024-08-19T01:30:51.552990+0000","last_unstale":"2024-08-19T01:33:13.820715+0000","last_undegraded":"2024-08-19T01:33:13.820715+0000","last_fullsized":"2024-08-19T01:33:13.820715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:28:33.193014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821634+0000","last_change":"2024-08-19T01:30:49.537742+0000","last_active":"2024-08-19T01:33:13.821634+0000","last_peered":"2024-08-19T01:33:13.821634+0000","last_clean":"2024-08-19T01:33:13.821634+0000","last_became_active":"2024-08-19T01:30:49.537153+0000","last_became_peered":"2024-08-19T01:30:49.537153+0000","last_unstale":"2024-08-19T01:33:13.821634+0000","last_undegraded":"2024-08-19T01:33:13.821634+0000","last_fullsized":"2024-08-19T01:33:13.821634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:25:03.650900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821745+0000","last_change":"2024-08-19T01:32:35.668059+0000","last_active":"2024-08-19T01:33:13.821745+0000","last_peered":"2024-08-19T01:33:13.821745+0000","last_clean":"2024-08-19T01:33:13.821745+0000","last_became_active":"2024-08-19T01:32:35.663768+0000","last_became_peered":"2024-08-19T01:32:35.663768+0000","last_unstale":"2024-08-19T01:33:13.821745+0000","last_undegraded":"2024-08-19T01:33:13.821745+0000","last_fullsized":"2024-08-19T01:33:13.821745+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:19:38.669783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009175+0000","last_change":"2024-08-19T01:30:51.552126+0000","last_active":"2024-08-19T01:33:14.009175+0000","last_peered":"2024-08-19T01:33:14.009175+0000","last_clean":"2024-08-19T01:33:14.009175+0000","last_became_active":"2024-08-19T01:30:51.551795+0000","last_became_peered":"2024-08-19T01:30:51.551795+0000","last_unstale":"2024-08-19T01:33:14.009175+0000","last_undegraded":"2024-08-19T01:33:14.009175+0000","last_fullsized":"2024-08-19T01:33:14.009175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:22:54.705359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820641+0000","last_change":"2024-08-19T01:30:49.536659+0000","last_active":"2024-08-19T01:33:13.820641+0000","last_peered":"2024-08-19T01:33:13.820641+0000","last_clean":"2024-08-19T01:33:13.820641+0000","last_became_active":"2024-08-19T01:30:49.536366+0000","last_became_peered":"2024-08-19T01:30:49.536366+0000","last_unstale":"2024-08-19T01:33:13.820641+0000","last_undegraded":"2024-08-19T01:33:13.820641+0000","last_fullsized":"2024-08-19T01:33:13.820641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:58:52.547461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916618+0000","last_change":"2024-08-19T01:30:58.593724+0000","last_active":"2024-08-19T01:33:13.916618+0000","last_peered":"2024-08-19T01:33:13.916618+0000","last_clean":"2024-08-19T01:33:13.916618+0000","last_became_active":"2024-08-19T01:30:58.593383+0000","last_became_peered":"2024-08-19T01:30:58.593383+0000","last_unstale":"2024-08-19T01:33:13.916618+0000","last_undegraded":"2024-08-19T01:33:13.916618+0000","last_fullsized":"2024-08-19T01:33:13.916618+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:54:26.424139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920243+0000","last_change":"2024-08-19T01:30:51.557682+0000","last_active":"2024-08-19T01:33:13.920243+0000","last_peered":"2024-08-19T01:33:13.920243+0000","last_clean":"2024-08-19T01:33:13.920243+0000","last_became_active":"2024-08-19T01:30:51.557351+0000","last_became_peered":"2024-08-19T01:30:51.557351+0000","last_unstale":"2024-08-19T01:33:13.920243+0000","last_undegraded":"2024-08-19T01:33:13.920243+0000","last_fullsized":"2024-08-19T01:33:13.920243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:41.484955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009059+0000","last_change":"2024-08-19T01:30:49.575996+0000","last_active":"2024-08-19T01:33:14.009059+0000","last_peered":"2024-08-19T01:33:14.009059+0000","last_clean":"2024-08-19T01:33:14.009059+0000","last_became_active":"2024-08-19T01:30:49.575607+0000","last_became_peered":"2024-08-19T01:30:49.575607+0000","last_unstale":"2024-08-19T01:33:14.009059+0000","last_undegraded":"2024-08-19T01:33:14.009059+0000","last_fullsized":"2024-08-19T01:33:14.009059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:20:33.467100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821854+0000","last_change":"2024-08-19T01:30:58.592629+0000","last_active":"2024-08-19T01:33:13.821854+0000","last_peered":"2024-08-19T01:33:13.821854+0000","last_clean":"2024-08-19T01:33:13.821854+0000","last_became_active":"2024-08-19T01:30:58.592122+0000","last_became_peered":"2024-08-19T01:30:58.592122+0000","last_unstale":"2024-08-19T01:33:13.821854+0000","last_undegraded":"2024-08-19T01:33:13.821854+0000","last_fullsized":"2024-08-19T01:33:13.821854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:19:09.846829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011482+0000","last_change":"2024-08-19T01:31:35.746416+0000","last_active":"2024-08-19T01:33:14.011482+0000","last_peered":"2024-08-19T01:33:14.011482+0000","last_clean":"2024-08-19T01:33:14.011482+0000","last_became_active":"2024-08-19T01:31:35.746110+0000","last_became_peered":"2024-08-19T01:31:35.746110+0000","last_unstale":"2024-08-19T01:33:14.011482+0000","last_undegraded":"2024-08-19T01:33:14.011482+0000","last_fullsized":"2024-08-19T01:33:14.011482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:29:29.789785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821949+0000","last_change":"2024-08-19T01:30:49.711366+0000","last_active":"2024-08-19T01:33:13.821949+0000","last_peered":"2024-08-19T01:33:13.821949+0000","last_clean":"2024-08-19T01:33:13.821949+0000","last_became_active":"2024-08-19T01:30:49.711078+0000","last_became_peered":"2024-08-19T01:30:49.711078+0000","last_unstale":"2024-08-19T01:33:13.821949+0000","last_undegraded":"2024-08-19T01:33:13.821949+0000","last_fullsized":"2024-08-19T01:33:13.821949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:11:16.767867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821530+0000","last_change":"2024-08-19T01:30:58.591845+0000","last_active":"2024-08-19T01:33:13.821530+0000","last_peered":"2024-08-19T01:33:13.821530+0000","last_clean":"2024-08-19T01:33:13.821530+0000","last_became_active":"2024-08-19T01:30:58.591453+0000","last_became_peered":"2024-08-19T01:30:58.591453+0000","last_unstale":"2024-08-19T01:33:13.821530+0000","last_undegraded":"2024-08-19T01:33:13.821530+0000","last_fullsized":"2024-08-19T01:33:13.821530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:37:09.521514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820925+0000","last_change":"2024-08-19T01:30:51.563156+0000","last_active":"2024-08-19T01:33:13.820925+0000","last_peered":"2024-08-19T01:33:13.820925+0000","last_clean":"2024-08-19T01:33:13.820925+0000","last_became_active":"2024-08-19T01:30:51.562860+0000","last_became_peered":"2024-08-19T01:30:51.562860+0000","last_unstale":"2024-08-19T01:33:13.820925+0000","last_undegraded":"2024-08-19T01:33:13.820925+0000","last_fullsized":"2024-08-19T01:33:13.820925+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:54:11.768833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":112,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.053205+0000","last_change":"2024-08-19T01:30:49.536648+0000","last_active":"2024-08-19T01:33:14.053205+0000","last_peered":"2024-08-19T01:33:14.053205+0000","last_clean":"2024-08-19T01:33:14.053205+0000","last_became_active":"2024-08-19T01:30:49.536080+0000","last_became_peered":"2024-08-19T01:30:49.536080+0000","last_unstale":"2024-08-19T01:33:14.053205+0000","last_undegraded":"2024-08-19T01:33:14.053205+0000","last_fullsized":"2024-08-19T01:33:14.053205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:03:46.804252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":87,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820301+0000","last_change":"2024-08-19T01:30:58.592253+0000","last_active":"2024-08-19T01:33:13.820301+0000","last_peered":"2024-08-19T01:33:13.820301+0000","last_clean":"2024-08-19T01:33:13.820301+0000","last_became_active":"2024-08-19T01:30:58.591911+0000","last_became_peered":"2024-08-19T01:30:58.591911+0000","last_unstale":"2024-08-19T01:33:13.820301+0000","last_undegraded":"2024-08-19T01:33:13.820301+0000","last_fullsized":"2024-08-19T01:33:13.820301+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:02:27.112597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010224+0000","last_change":"2024-08-19T01:30:51.547607+0000","last_active":"2024-08-19T01:33:14.010224+0000","last_peered":"2024-08-19T01:33:14.010224+0000","last_clean":"2024-08-19T01:33:14.010224+0000","last_became_active":"2024-08-19T01:30:51.547316+0000","last_became_peered":"2024-08-19T01:30:51.547316+0000","last_unstale":"2024-08-19T01:33:14.010224+0000","last_undegraded":"2024-08-19T01:33:14.010224+0000","last_fullsized":"2024-08-19T01:33:14.010224+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:48:58.300508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920697+0000","last_change":"2024-08-19T01:30:49.576942+0000","last_active":"2024-08-19T01:33:13.920697+0000","last_peered":"2024-08-19T01:33:13.920697+0000","last_clean":"2024-08-19T01:33:13.920697+0000","last_became_active":"2024-08-19T01:30:49.576528+0000","last_became_peered":"2024-08-19T01:30:49.576528+0000","last_unstale":"2024-08-19T01:33:13.920697+0000","last_undegraded":"2024-08-19T01:33:13.920697+0000","last_fullsized":"2024-08-19T01:33:13.920697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:23:53.259395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009281+0000","last_change":"2024-08-19T01:30:58.591577+0000","last_active":"2024-08-19T01:33:14.009281+0000","last_peered":"2024-08-19T01:33:14.009281+0000","last_clean":"2024-08-19T01:33:14.009281+0000","last_became_active":"2024-08-19T01:30:58.591181+0000","last_became_peered":"2024-08-19T01:30:58.591181+0000","last_unstale":"2024-08-19T01:33:14.009281+0000","last_undegraded":"2024-08-19T01:33:14.009281+0000","last_fullsized":"2024-08-19T01:33:14.009281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:02:59.682147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932531+0000","last_change":"2024-08-19T01:30:51.548556+0000","last_active":"2024-08-19T01:33:13.932531+0000","last_peered":"2024-08-19T01:33:13.932531+0000","last_clean":"2024-08-19T01:33:13.932531+0000","last_became_active":"2024-08-19T01:30:51.547838+0000","last_became_peered":"2024-08-19T01:30:51.547838+0000","last_unstale":"2024-08-19T01:33:13.932531+0000","last_undegraded":"2024-08-19T01:33:13.932531+0000","last_fullsized":"2024-08-19T01:33:13.932531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:08:49.671964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916085+0000","last_change":"2024-08-19T01:30:49.576799+0000","last_active":"2024-08-19T01:33:13.916085+0000","last_peered":"2024-08-19T01:33:13.916085+0000","last_clean":"2024-08-19T01:33:13.916085+0000","last_became_active":"2024-08-19T01:30:49.576385+0000","last_became_peered":"2024-08-19T01:30:49.576385+0000","last_unstale":"2024-08-19T01:33:13.916085+0000","last_undegraded":"2024-08-19T01:33:13.916085+0000","last_fullsized":"2024-08-19T01:33:13.916085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:34:18.287518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920261+0000","last_change":"2024-08-19T01:30:58.593703+0000","last_active":"2024-08-19T01:33:13.920261+0000","last_peered":"2024-08-19T01:33:13.920261+0000","last_clean":"2024-08-19T01:33:13.920261+0000","last_became_active":"2024-08-19T01:30:58.593303+0000","last_became_peered":"2024-08-19T01:30:58.593303+0000","last_unstale":"2024-08-19T01:33:13.920261+0000","last_undegraded":"2024-08-19T01:33:13.920261+0000","last_fullsized":"2024-08-19T01:33:13.920261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:12:43.359262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932455+0000","last_change":"2024-08-19T01:31:35.601173+0000","last_active":"2024-08-19T01:33:13.932455+0000","last_peered":"2024-08-19T01:33:13.932455+0000","last_clean":"2024-08-19T01:33:13.932455+0000","last_became_active":"2024-08-19T01:31:35.600742+0000","last_became_peered":"2024-08-19T01:31:35.600742+0000","last_unstale":"2024-08-19T01:33:13.932455+0000","last_undegraded":"2024-08-19T01:33:13.932455+0000","last_fullsized":"2024-08-19T01:33:13.932455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:36:48.977124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009550+0000","last_change":"2024-08-19T01:30:49.540070+0000","last_active":"2024-08-19T01:33:14.009550+0000","last_peered":"2024-08-19T01:33:14.009550+0000","last_clean":"2024-08-19T01:33:14.009550+0000","last_became_active":"2024-08-19T01:30:49.539706+0000","last_became_peered":"2024-08-19T01:30:49.539706+0000","last_unstale":"2024-08-19T01:33:14.009550+0000","last_undegraded":"2024-08-19T01:33:14.009550+0000","last_fullsized":"2024-08-19T01:33:14.009550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:14:15.813670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009939+0000","last_change":"2024-08-19T01:30:58.590766+0000","last_active":"2024-08-19T01:33:14.009939+0000","last_peered":"2024-08-19T01:33:14.009939+0000","last_clean":"2024-08-19T01:33:14.009939+0000","last_became_active":"2024-08-19T01:30:58.590415+0000","last_became_peered":"2024-08-19T01:30:58.590415+0000","last_unstale":"2024-08-19T01:33:14.009939+0000","last_undegraded":"2024-08-19T01:33:14.009939+0000","last_fullsized":"2024-08-19T01:33:14.009939+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:48:59.261086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914324+0000","last_change":"2024-08-19T01:31:35.602332+0000","last_active":"2024-08-19T01:33:13.914324+0000","last_peered":"2024-08-19T01:33:13.914324+0000","last_clean":"2024-08-19T01:33:13.914324+0000","last_became_active":"2024-08-19T01:31:35.602007+0000","last_became_peered":"2024-08-19T01:31:35.602007+0000","last_unstale":"2024-08-19T01:33:13.914324+0000","last_undegraded":"2024-08-19T01:33:13.914324+0000","last_fullsized":"2024-08-19T01:33:13.914324+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:17:47.694567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":153,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:18.743740+0000","last_change":"2024-08-19T01:30:49.575995+0000","last_active":"2024-08-19T01:33:18.743740+0000","last_peered":"2024-08-19T01:33:18.743740+0000","last_clean":"2024-08-19T01:33:18.743740+0000","last_became_active":"2024-08-19T01:30:49.575249+0000","last_became_peered":"2024-08-19T01:30:49.575249+0000","last_unstale":"2024-08-19T01:33:18.743740+0000","last_undegraded":"2024-08-19T01:33:18.743740+0000","last_fullsized":"2024-08-19T01:33:18.743740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:39:05.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011011+0000","last_change":"2024-08-19T01:32:35.725750+0000","last_active":"2024-08-19T01:33:14.011011+0000","last_peered":"2024-08-19T01:33:14.011011+0000","last_clean":"2024-08-19T01:33:14.011011+0000","last_became_active":"2024-08-19T01:32:35.725320+0000","last_became_peered":"2024-08-19T01:32:35.725320+0000","last_unstale":"2024-08-19T01:33:14.011011+0000","last_undegraded":"2024-08-19T01:33:14.011011+0000","last_fullsized":"2024-08-19T01:33:14.011011+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:10.492801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009295+0000","last_change":"2024-08-19T01:30:51.549158+0000","last_active":"2024-08-19T01:33:14.009295+0000","last_peered":"2024-08-19T01:33:14.009295+0000","last_clean":"2024-08-19T01:33:14.009295+0000","last_became_active":"2024-08-19T01:30:51.548854+0000","last_became_peered":"2024-08-19T01:30:51.548854+0000","last_unstale":"2024-08-19T01:33:14.009295+0000","last_undegraded":"2024-08-19T01:33:14.009295+0000","last_fullsized":"2024-08-19T01:33:14.009295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:35.661758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009708+0000","last_change":"2024-08-19T01:30:49.538282+0000","last_active":"2024-08-19T01:33:14.009708+0000","last_peered":"2024-08-19T01:33:14.009708+0000","last_clean":"2024-08-19T01:33:14.009708+0000","last_became_active":"2024-08-19T01:30:49.537918+0000","last_became_peered":"2024-08-19T01:30:49.537918+0000","last_unstale":"2024-08-19T01:33:14.009708+0000","last_undegraded":"2024-08-19T01:33:14.009708+0000","last_fullsized":"2024-08-19T01:33:14.009708+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:27.886958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929499+0000","last_change":"2024-08-19T01:30:58.591153+0000","last_active":"2024-08-19T01:33:13.929499+0000","last_peered":"2024-08-19T01:33:13.929499+0000","last_clean":"2024-08-19T01:33:13.929499+0000","last_became_active":"2024-08-19T01:30:58.590738+0000","last_became_peered":"2024-08-19T01:30:58.590738+0000","last_unstale":"2024-08-19T01:33:13.929499+0000","last_undegraded":"2024-08-19T01:33:13.929499+0000","last_fullsized":"2024-08-19T01:33:13.929499+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:29.073825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820667+0000","last_change":"2024-08-19T01:30:51.554878+0000","last_active":"2024-08-19T01:33:13.820667+0000","last_peered":"2024-08-19T01:33:13.820667+0000","last_clean":"2024-08-19T01:33:13.820667+0000","last_became_active":"2024-08-19T01:30:51.553743+0000","last_became_peered":"2024-08-19T01:30:51.553743+0000","last_unstale":"2024-08-19T01:33:13.820667+0000","last_undegraded":"2024-08-19T01:33:13.820667+0000","last_fullsized":"2024-08-19T01:33:13.820667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:08:41.932641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822048+0000","last_change":"2024-08-19T01:30:49.542043+0000","last_active":"2024-08-19T01:33:13.822048+0000","last_peered":"2024-08-19T01:33:13.822048+0000","last_clean":"2024-08-19T01:33:13.822048+0000","last_became_active":"2024-08-19T01:30:49.541551+0000","last_became_peered":"2024-08-19T01:30:49.541551+0000","last_unstale":"2024-08-19T01:33:13.822048+0000","last_undegraded":"2024-08-19T01:33:13.822048+0000","last_fullsized":"2024-08-19T01:33:13.822048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:46:14.781516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":371,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009819+0000","last_change":"2024-08-19T01:29:41.574391+0000","last_active":"2024-08-19T01:33:14.009819+0000","last_peered":"2024-08-19T01:33:14.009819+0000","last_clean":"2024-08-19T01:33:14.009819+0000","last_became_active":"2024-08-19T01:29:41.573885+0000","last_became_peered":"2024-08-19T01:29:41.573885+0000","last_unstale":"2024-08-19T01:33:14.009819+0000","last_undegraded":"2024-08-19T01:33:14.009819+0000","last_fullsized":"2024-08-19T01:33:14.009819+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_clean_scrub_stamp":"2024-08-19T01:26:37.926225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:55:38.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010021+0000","last_change":"2024-08-19T01:30:58.598415+0000","last_active":"2024-08-19T01:33:14.010021+0000","last_peered":"2024-08-19T01:33:14.010021+0000","last_clean":"2024-08-19T01:33:14.010021+0000","last_became_active":"2024-08-19T01:30:58.598086+0000","last_became_peered":"2024-08-19T01:30:58.598086+0000","last_unstale":"2024-08-19T01:33:14.010021+0000","last_undegraded":"2024-08-19T01:33:14.010021+0000","last_fullsized":"2024-08-19T01:33:14.010021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:21:37.338869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010133+0000","last_change":"2024-08-19T01:30:51.560407+0000","last_active":"2024-08-19T01:33:14.010133+0000","last_peered":"2024-08-19T01:33:14.010133+0000","last_clean":"2024-08-19T01:33:14.010133+0000","last_became_active":"2024-08-19T01:30:51.559881+0000","last_became_peered":"2024-08-19T01:30:51.559881+0000","last_unstale":"2024-08-19T01:33:14.010133+0000","last_undegraded":"2024-08-19T01:33:14.010133+0000","last_fullsized":"2024-08-19T01:33:14.010133+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:24:37.928254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822063+0000","last_change":"2024-08-19T01:30:49.711171+0000","last_active":"2024-08-19T01:33:13.822063+0000","last_peered":"2024-08-19T01:33:13.822063+0000","last_clean":"2024-08-19T01:33:13.822063+0000","last_became_active":"2024-08-19T01:30:49.710707+0000","last_became_peered":"2024-08-19T01:30:49.710707+0000","last_unstale":"2024-08-19T01:33:13.822063+0000","last_undegraded":"2024-08-19T01:33:13.822063+0000","last_fullsized":"2024-08-19T01:33:13.822063+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:58:20.924502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009117+0000","last_change":"2024-08-19T01:30:58.590829+0000","last_active":"2024-08-19T01:33:14.009117+0000","last_peered":"2024-08-19T01:33:14.009117+0000","last_clean":"2024-08-19T01:33:14.009117+0000","last_became_active":"2024-08-19T01:30:58.590460+0000","last_became_peered":"2024-08-19T01:30:58.590460+0000","last_unstale":"2024-08-19T01:33:14.009117+0000","last_undegraded":"2024-08-19T01:33:14.009117+0000","last_fullsized":"2024-08-19T01:33:14.009117+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:04.118550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009951+0000","last_change":"2024-08-19T01:30:51.564990+0000","last_active":"2024-08-19T01:33:14.009951+0000","last_peered":"2024-08-19T01:33:14.009951+0000","last_clean":"2024-08-19T01:33:14.009951+0000","last_became_active":"2024-08-19T01:30:51.564653+0000","last_became_peered":"2024-08-19T01:30:51.564653+0000","last_unstale":"2024-08-19T01:33:14.009951+0000","last_undegraded":"2024-08-19T01:33:14.009951+0000","last_fullsized":"2024-08-19T01:33:14.009951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:15:41.154376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.046623+0000","last_change":"2024-08-19T01:30:49.537637+0000","last_active":"2024-08-19T01:33:14.046623+0000","last_peered":"2024-08-19T01:33:14.046623+0000","last_clean":"2024-08-19T01:33:14.046623+0000","last_became_active":"2024-08-19T01:30:49.537277+0000","last_became_peered":"2024-08-19T01:30:49.537277+0000","last_unstale":"2024-08-19T01:33:14.046623+0000","last_undegraded":"2024-08-19T01:33:14.046623+0000","last_fullsized":"2024-08-19T01:33:14.046623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:13.547597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010579+0000","last_change":"2024-08-19T01:30:58.594675+0000","last_active":"2024-08-19T01:33:14.010579+0000","last_peered":"2024-08-19T01:33:14.010579+0000","last_clean":"2024-08-19T01:33:14.010579+0000","last_became_active":"2024-08-19T01:30:58.594323+0000","last_became_peered":"2024-08-19T01:30:58.594323+0000","last_unstale":"2024-08-19T01:33:14.010579+0000","last_undegraded":"2024-08-19T01:33:14.010579+0000","last_fullsized":"2024-08-19T01:33:14.010579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:21:05.447106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010126+0000","last_change":"2024-08-19T01:30:51.565014+0000","last_active":"2024-08-19T01:33:14.010126+0000","last_peered":"2024-08-19T01:33:14.010126+0000","last_clean":"2024-08-19T01:33:14.010126+0000","last_became_active":"2024-08-19T01:30:51.564672+0000","last_became_peered":"2024-08-19T01:30:51.564672+0000","last_unstale":"2024-08-19T01:33:14.010126+0000","last_undegraded":"2024-08-19T01:33:14.010126+0000","last_fullsized":"2024-08-19T01:33:14.010126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:56:07.404042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010841+0000","last_change":"2024-08-19T01:30:49.538423+0000","last_active":"2024-08-19T01:33:14.010841+0000","last_peered":"2024-08-19T01:33:14.010841+0000","last_clean":"2024-08-19T01:33:14.010841+0000","last_became_active":"2024-08-19T01:30:49.538049+0000","last_became_peered":"2024-08-19T01:30:49.538049+0000","last_unstale":"2024-08-19T01:33:14.010841+0000","last_undegraded":"2024-08-19T01:33:14.010841+0000","last_fullsized":"2024-08-19T01:33:14.010841+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:27:41.449742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929000+0000","last_change":"2024-08-19T01:30:58.590963+0000","last_active":"2024-08-19T01:33:13.929000+0000","last_peered":"2024-08-19T01:33:13.929000+0000","last_clean":"2024-08-19T01:33:13.929000+0000","last_became_active":"2024-08-19T01:30:58.590584+0000","last_became_peered":"2024-08-19T01:30:58.590584+0000","last_unstale":"2024-08-19T01:33:13.929000+0000","last_undegraded":"2024-08-19T01:33:13.929000+0000","last_fullsized":"2024-08-19T01:33:13.929000+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:37.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009458+0000","last_change":"2024-08-19T01:30:51.561298+0000","last_active":"2024-08-19T01:33:14.009458+0000","last_peered":"2024-08-19T01:33:14.009458+0000","last_clean":"2024-08-19T01:33:14.009458+0000","last_became_active":"2024-08-19T01:30:51.560919+0000","last_became_peered":"2024-08-19T01:30:51.560919+0000","last_unstale":"2024-08-19T01:33:14.009458+0000","last_undegraded":"2024-08-19T01:33:14.009458+0000","last_fullsized":"2024-08-19T01:33:14.009458+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:04:39.808838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920821+0000","last_change":"2024-08-19T01:30:49.541968+0000","last_active":"2024-08-19T01:33:13.920821+0000","last_peered":"2024-08-19T01:33:13.920821+0000","last_clean":"2024-08-19T01:33:13.920821+0000","last_became_active":"2024-08-19T01:30:49.541512+0000","last_became_peered":"2024-08-19T01:30:49.541512+0000","last_unstale":"2024-08-19T01:33:13.920821+0000","last_undegraded":"2024-08-19T01:33:13.920821+0000","last_fullsized":"2024-08-19T01:33:13.920821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:34:28.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929302+0000","last_change":"2024-08-19T01:30:58.594776+0000","last_active":"2024-08-19T01:33:13.929302+0000","last_peered":"2024-08-19T01:33:13.929302+0000","last_clean":"2024-08-19T01:33:13.929302+0000","last_became_active":"2024-08-19T01:30:58.594414+0000","last_became_peered":"2024-08-19T01:30:58.594414+0000","last_unstale":"2024-08-19T01:33:13.929302+0000","last_undegraded":"2024-08-19T01:33:13.929302+0000","last_fullsized":"2024-08-19T01:33:13.929302+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:32:20.121504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929211+0000","last_change":"2024-08-19T01:30:51.556696+0000","last_active":"2024-08-19T01:33:13.929211+0000","last_peered":"2024-08-19T01:33:13.929211+0000","last_clean":"2024-08-19T01:33:13.929211+0000","last_became_active":"2024-08-19T01:30:51.555873+0000","last_became_peered":"2024-08-19T01:30:51.555873+0000","last_unstale":"2024-08-19T01:33:13.929211+0000","last_undegraded":"2024-08-19T01:33:13.929211+0000","last_fullsized":"2024-08-19T01:33:13.929211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:50:27.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920308+0000","last_change":"2024-08-19T01:30:49.576925+0000","last_active":"2024-08-19T01:33:13.920308+0000","last_peered":"2024-08-19T01:33:13.920308+0000","last_clean":"2024-08-19T01:33:13.920308+0000","last_became_active":"2024-08-19T01:30:49.576509+0000","last_became_peered":"2024-08-19T01:30:49.576509+0000","last_unstale":"2024-08-19T01:33:13.920308+0000","last_undegraded":"2024-08-19T01:33:13.920308+0000","last_fullsized":"2024-08-19T01:33:13.920308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:43:01.887020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010743+0000","last_change":"2024-08-19T01:32:35.826488+0000","last_active":"2024-08-19T01:33:14.010743+0000","last_peered":"2024-08-19T01:33:14.010743+0000","last_clean":"2024-08-19T01:33:14.010743+0000","last_became_active":"2024-08-19T01:32:35.826036+0000","last_became_peered":"2024-08-19T01:32:35.826036+0000","last_unstale":"2024-08-19T01:33:14.010743+0000","last_undegraded":"2024-08-19T01:33:14.010743+0000","last_fullsized":"2024-08-19T01:33:14.010743+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:16:03.388117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009567+0000","last_change":"2024-08-19T01:30:51.559622+0000","last_active":"2024-08-19T01:33:14.009567+0000","last_peered":"2024-08-19T01:33:14.009567+0000","last_clean":"2024-08-19T01:33:14.009567+0000","last_became_active":"2024-08-19T01:30:51.559263+0000","last_became_peered":"2024-08-19T01:30:51.559263+0000","last_unstale":"2024-08-19T01:33:14.009567+0000","last_undegraded":"2024-08-19T01:33:14.009567+0000","last_fullsized":"2024-08-19T01:33:14.009567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:11:25.758915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":155,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.739646+0000","last_change":"2024-08-19T01:30:49.540102+0000","last_active":"2024-08-19T01:33:14.739646+0000","last_peered":"2024-08-19T01:33:14.739646+0000","last_clean":"2024-08-19T01:33:14.739646+0000","last_became_active":"2024-08-19T01:30:49.539740+0000","last_became_peered":"2024-08-19T01:30:49.539740+0000","last_unstale":"2024-08-19T01:33:14.739646+0000","last_undegraded":"2024-08-19T01:33:14.739646+0000","last_fullsized":"2024-08-19T01:33:14.739646+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:47:21.684925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1899,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916713+0000","last_change":"2024-08-19T01:30:58.595748+0000","last_active":"2024-08-19T01:33:13.916713+0000","last_peered":"2024-08-19T01:33:13.916713+0000","last_clean":"2024-08-19T01:33:13.916713+0000","last_became_active":"2024-08-19T01:30:58.595396+0000","last_became_peered":"2024-08-19T01:30:58.595396+0000","last_unstale":"2024-08-19T01:33:13.916713+0000","last_undegraded":"2024-08-19T01:33:13.916713+0000","last_fullsized":"2024-08-19T01:33:13.916713+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:58:29.113687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009400+0000","last_change":"2024-08-19T01:30:51.559159+0000","last_active":"2024-08-19T01:33:14.009400+0000","last_peered":"2024-08-19T01:33:14.009400+0000","last_clean":"2024-08-19T01:33:14.009400+0000","last_became_active":"2024-08-19T01:30:51.558226+0000","last_became_peered":"2024-08-19T01:30:51.558226+0000","last_unstale":"2024-08-19T01:33:14.009400+0000","last_undegraded":"2024-08-19T01:33:14.009400+0000","last_fullsized":"2024-08-19T01:33:14.009400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:12:54.447301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929394+0000","last_change":"2024-08-19T01:30:49.540806+0000","last_active":"2024-08-19T01:33:13.929394+0000","last_peered":"2024-08-19T01:33:13.929394+0000","last_clean":"2024-08-19T01:33:13.929394+0000","last_became_active":"2024-08-19T01:30:49.540439+0000","last_became_peered":"2024-08-19T01:30:49.540439+0000","last_unstale":"2024-08-19T01:33:13.929394+0000","last_undegraded":"2024-08-19T01:33:13.929394+0000","last_fullsized":"2024-08-19T01:33:13.929394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:40.088842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009235+0000","last_change":"2024-08-19T01:30:58.592661+0000","last_active":"2024-08-19T01:33:14.009235+0000","last_peered":"2024-08-19T01:33:14.009235+0000","last_clean":"2024-08-19T01:33:14.009235+0000","last_became_active":"2024-08-19T01:30:58.592066+0000","last_became_peered":"2024-08-19T01:30:58.592066+0000","last_unstale":"2024-08-19T01:33:14.009235+0000","last_undegraded":"2024-08-19T01:33:14.009235+0000","last_fullsized":"2024-08-19T01:33:14.009235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:35.039682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009768+0000","last_change":"2024-08-19T01:30:51.558497+0000","last_active":"2024-08-19T01:33:14.009768+0000","last_peered":"2024-08-19T01:33:14.009768+0000","last_clean":"2024-08-19T01:33:14.009768+0000","last_became_active":"2024-08-19T01:30:51.558185+0000","last_became_peered":"2024-08-19T01:30:51.558185+0000","last_unstale":"2024-08-19T01:33:14.009768+0000","last_undegraded":"2024-08-19T01:33:14.009768+0000","last_fullsized":"2024-08-19T01:33:14.009768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:30:42.766269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821845+0000","last_change":"2024-08-19T01:30:49.539112+0000","last_active":"2024-08-19T01:33:13.821845+0000","last_peered":"2024-08-19T01:33:13.821845+0000","last_clean":"2024-08-19T01:33:13.821845+0000","last_became_active":"2024-08-19T01:30:49.538798+0000","last_became_peered":"2024-08-19T01:30:49.538798+0000","last_unstale":"2024-08-19T01:33:13.821845+0000","last_undegraded":"2024-08-19T01:33:13.821845+0000","last_fullsized":"2024-08-19T01:33:13.821845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:10:58.977041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929436+0000","last_change":"2024-08-19T01:30:58.590727+0000","last_active":"2024-08-19T01:33:13.929436+0000","last_peered":"2024-08-19T01:33:13.929436+0000","last_clean":"2024-08-19T01:33:13.929436+0000","last_became_active":"2024-08-19T01:30:58.590208+0000","last_became_peered":"2024-08-19T01:30:58.590208+0000","last_unstale":"2024-08-19T01:33:13.929436+0000","last_undegraded":"2024-08-19T01:33:13.929436+0000","last_fullsized":"2024-08-19T01:33:13.929436+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:36.422915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920646+0000","last_change":"2024-08-19T01:30:51.565113+0000","last_active":"2024-08-19T01:33:13.920646+0000","last_peered":"2024-08-19T01:33:13.920646+0000","last_clean":"2024-08-19T01:33:13.920646+0000","last_became_active":"2024-08-19T01:30:51.564769+0000","last_became_peered":"2024-08-19T01:30:51.564769+0000","last_unstale":"2024-08-19T01:33:13.920646+0000","last_undegraded":"2024-08-19T01:33:13.920646+0000","last_fullsized":"2024-08-19T01:33:13.920646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:13:28.373114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":311,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.739508+0000","last_change":"2024-08-19T01:31:35.610319+0000","last_active":"2024-08-19T01:33:14.739508+0000","last_peered":"2024-08-19T01:33:14.739508+0000","last_clean":"2024-08-19T01:33:14.739508+0000","last_became_active":"2024-08-19T01:31:35.602527+0000","last_became_peered":"2024-08-19T01:31:35.602527+0000","last_unstale":"2024-08-19T01:33:14.739508+0000","last_undegraded":"2024-08-19T01:33:14.739508+0000","last_fullsized":"2024-08-19T01:33:14.739508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:50:05.511332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68780,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":210,"num_read_kb":170,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3841,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.915346+0000","last_change":"2024-08-19T01:30:58.594836+0000","last_active":"2024-08-19T01:33:13.915346+0000","last_peered":"2024-08-19T01:33:13.915346+0000","last_clean":"2024-08-19T01:33:13.915346+0000","last_became_active":"2024-08-19T01:30:58.594441+0000","last_became_peered":"2024-08-19T01:30:58.594441+0000","last_unstale":"2024-08-19T01:33:13.915346+0000","last_undegraded":"2024-08-19T01:33:13.915346+0000","last_fullsized":"2024-08-19T01:33:13.915346+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:29:16.482181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009266+0000","last_change":"2024-08-19T01:30:51.562567+0000","last_active":"2024-08-19T01:33:14.009266+0000","last_peered":"2024-08-19T01:33:14.009266+0000","last_clean":"2024-08-19T01:33:14.009266+0000","last_became_active":"2024-08-19T01:30:51.562211+0000","last_became_peered":"2024-08-19T01:30:51.562211+0000","last_unstale":"2024-08-19T01:33:14.009266+0000","last_undegraded":"2024-08-19T01:33:14.009266+0000","last_fullsized":"2024-08-19T01:33:14.009266+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:58.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929607+0000","last_change":"2024-08-19T01:30:49.575957+0000","last_active":"2024-08-19T01:33:13.929607+0000","last_peered":"2024-08-19T01:33:13.929607+0000","last_clean":"2024-08-19T01:33:13.929607+0000","last_became_active":"2024-08-19T01:30:49.575570+0000","last_became_peered":"2024-08-19T01:30:49.575570+0000","last_unstale":"2024-08-19T01:33:13.929607+0000","last_undegraded":"2024-08-19T01:33:13.929607+0000","last_fullsized":"2024-08-19T01:33:13.929607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:47.038113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821520+0000","last_change":"2024-08-19T01:32:35.826212+0000","last_active":"2024-08-19T01:33:13.821520+0000","last_peered":"2024-08-19T01:33:13.821520+0000","last_clean":"2024-08-19T01:33:13.821520+0000","last_became_active":"2024-08-19T01:32:35.825748+0000","last_became_peered":"2024-08-19T01:32:35.825748+0000","last_unstale":"2024-08-19T01:33:13.821520+0000","last_undegraded":"2024-08-19T01:33:13.821520+0000","last_fullsized":"2024-08-19T01:33:13.821520+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:34:59.072131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011949+0000","last_change":"2024-08-19T01:31:35.606449+0000","last_active":"2024-08-19T01:33:14.011949+0000","last_peered":"2024-08-19T01:33:14.011949+0000","last_clean":"2024-08-19T01:33:14.011949+0000","last_became_active":"2024-08-19T01:31:35.606105+0000","last_became_peered":"2024-08-19T01:31:35.606105+0000","last_unstale":"2024-08-19T01:33:14.011949+0000","last_undegraded":"2024-08-19T01:33:14.011949+0000","last_fullsized":"2024-08-19T01:33:14.011949+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:36:21.834763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010024+0000","last_change":"2024-08-19T01:30:49.542441+0000","last_active":"2024-08-19T01:33:14.010024+0000","last_peered":"2024-08-19T01:33:14.010024+0000","last_clean":"2024-08-19T01:33:14.010024+0000","last_became_active":"2024-08-19T01:30:49.542106+0000","last_became_peered":"2024-08-19T01:30:49.542106+0000","last_unstale":"2024-08-19T01:33:14.010024+0000","last_undegraded":"2024-08-19T01:33:14.010024+0000","last_fullsized":"2024-08-19T01:33:14.010024+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:21:01.956282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914501+0000","last_change":"2024-08-19T01:30:58.588899+0000","last_active":"2024-08-19T01:33:13.914501+0000","last_peered":"2024-08-19T01:33:13.914501+0000","last_clean":"2024-08-19T01:33:13.914501+0000","last_became_active":"2024-08-19T01:30:58.588521+0000","last_became_peered":"2024-08-19T01:30:58.588521+0000","last_unstale":"2024-08-19T01:33:13.914501+0000","last_undegraded":"2024-08-19T01:33:13.914501+0000","last_fullsized":"2024-08-19T01:33:13.914501+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:43:59.485456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009765+0000","last_change":"2024-08-19T01:30:51.557248+0000","last_active":"2024-08-19T01:33:14.009765+0000","last_peered":"2024-08-19T01:33:14.009765+0000","last_clean":"2024-08-19T01:33:14.009765+0000","last_became_active":"2024-08-19T01:30:51.556448+0000","last_became_peered":"2024-08-19T01:30:51.556448+0000","last_unstale":"2024-08-19T01:33:14.009765+0000","last_undegraded":"2024-08-19T01:33:14.009765+0000","last_fullsized":"2024-08-19T01:33:14.009765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:19:19.587955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":145,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932795+0000","last_change":"2024-08-19T01:30:49.540360+0000","last_active":"2024-08-19T01:33:13.932795+0000","last_peered":"2024-08-19T01:33:13.932795+0000","last_clean":"2024-08-19T01:33:13.932795+0000","last_became_active":"2024-08-19T01:30:49.539993+0000","last_became_peered":"2024-08-19T01:30:49.539993+0000","last_unstale":"2024-08-19T01:33:13.932795+0000","last_undegraded":"2024-08-19T01:33:13.932795+0000","last_fullsized":"2024-08-19T01:33:13.932795+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:05:28.308441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820478+0000","last_change":"2024-08-19T01:30:58.592387+0000","last_active":"2024-08-19T01:33:13.820478+0000","last_peered":"2024-08-19T01:33:13.820478+0000","last_clean":"2024-08-19T01:33:13.820478+0000","last_became_active":"2024-08-19T01:30:58.592031+0000","last_became_peered":"2024-08-19T01:30:58.592031+0000","last_unstale":"2024-08-19T01:33:13.820478+0000","last_undegraded":"2024-08-19T01:33:13.820478+0000","last_fullsized":"2024-08-19T01:33:13.820478+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:31:23.159012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011277+0000","last_change":"2024-08-19T01:30:58.595650+0000","last_active":"2024-08-19T01:33:14.011277+0000","last_peered":"2024-08-19T01:33:14.011277+0000","last_clean":"2024-08-19T01:33:14.011277+0000","last_became_active":"2024-08-19T01:30:58.595304+0000","last_became_peered":"2024-08-19T01:30:58.595304+0000","last_unstale":"2024-08-19T01:33:14.011277+0000","last_undegraded":"2024-08-19T01:33:14.011277+0000","last_fullsized":"2024-08-19T01:33:14.011277+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:23:53.517355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914930+0000","last_change":"2024-08-19T01:30:51.548683+0000","last_active":"2024-08-19T01:33:13.914930+0000","last_peered":"2024-08-19T01:33:13.914930+0000","last_clean":"2024-08-19T01:33:13.914930+0000","last_became_active":"2024-08-19T01:30:51.548382+0000","last_became_peered":"2024-08-19T01:30:51.548382+0000","last_unstale":"2024-08-19T01:33:13.914930+0000","last_undegraded":"2024-08-19T01:33:13.914930+0000","last_fullsized":"2024-08-19T01:33:13.914930+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:46:37.551017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929191+0000","last_change":"2024-08-19T01:30:49.540136+0000","last_active":"2024-08-19T01:33:13.929191+0000","last_peered":"2024-08-19T01:33:13.929191+0000","last_clean":"2024-08-19T01:33:13.929191+0000","last_became_active":"2024-08-19T01:30:49.539680+0000","last_became_peered":"2024-08-19T01:30:49.539680+0000","last_unstale":"2024-08-19T01:33:13.929191+0000","last_undegraded":"2024-08-19T01:33:13.929191+0000","last_fullsized":"2024-08-19T01:33:13.929191+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:25.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009451+0000","last_change":"2024-08-19T01:30:58.597130+0000","last_active":"2024-08-19T01:33:14.009451+0000","last_peered":"2024-08-19T01:33:14.009451+0000","last_clean":"2024-08-19T01:33:14.009451+0000","last_became_active":"2024-08-19T01:30:58.596834+0000","last_became_peered":"2024-08-19T01:30:58.596834+0000","last_unstale":"2024-08-19T01:33:14.009451+0000","last_undegraded":"2024-08-19T01:33:14.009451+0000","last_fullsized":"2024-08-19T01:33:14.009451+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:13:10.441421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820451+0000","last_change":"2024-08-19T01:30:51.555575+0000","last_active":"2024-08-19T01:33:13.820451+0000","last_peered":"2024-08-19T01:33:13.820451+0000","last_clean":"2024-08-19T01:33:13.820451+0000","last_became_active":"2024-08-19T01:30:51.555309+0000","last_became_peered":"2024-08-19T01:30:51.555309+0000","last_unstale":"2024-08-19T01:33:13.820451+0000","last_undegraded":"2024-08-19T01:33:13.820451+0000","last_fullsized":"2024-08-19T01:33:13.820451+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:11:25.515286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009505+0000","last_change":"2024-08-19T01:30:49.536844+0000","last_active":"2024-08-19T01:33:14.009505+0000","last_peered":"2024-08-19T01:33:14.009505+0000","last_clean":"2024-08-19T01:33:14.009505+0000","last_became_active":"2024-08-19T01:30:49.536496+0000","last_became_peered":"2024-08-19T01:30:49.536496+0000","last_unstale":"2024-08-19T01:33:14.009505+0000","last_undegraded":"2024-08-19T01:33:14.009505+0000","last_fullsized":"2024-08-19T01:33:14.009505+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:04.917188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932414+0000","last_change":"2024-08-19T01:30:49.539356+0000","last_active":"2024-08-19T01:33:13.932414+0000","last_peered":"2024-08-19T01:33:13.932414+0000","last_clean":"2024-08-19T01:33:13.932414+0000","last_became_active":"2024-08-19T01:30:49.538953+0000","last_became_peered":"2024-08-19T01:30:49.538953+0000","last_unstale":"2024-08-19T01:33:13.932414+0000","last_undegraded":"2024-08-19T01:33:13.932414+0000","last_fullsized":"2024-08-19T01:33:13.932414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:13:17.845140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929202+0000","last_change":"2024-08-19T01:31:35.870021+0000","last_active":"2024-08-19T01:33:13.929202+0000","last_peered":"2024-08-19T01:33:13.929202+0000","last_clean":"2024-08-19T01:33:13.929202+0000","last_became_active":"2024-08-19T01:31:35.869742+0000","last_became_peered":"2024-08-19T01:31:35.869742+0000","last_unstale":"2024-08-19T01:33:13.929202+0000","last_undegraded":"2024-08-19T01:33:13.929202+0000","last_fullsized":"2024-08-19T01:33:13.929202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:51.848694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821878+0000","last_change":"2024-08-19T01:30:58.590385+0000","last_active":"2024-08-19T01:33:13.821878+0000","last_peered":"2024-08-19T01:33:13.821878+0000","last_clean":"2024-08-19T01:33:13.821878+0000","last_became_active":"2024-08-19T01:30:58.590004+0000","last_became_peered":"2024-08-19T01:30:58.590004+0000","last_unstale":"2024-08-19T01:33:13.821878+0000","last_undegraded":"2024-08-19T01:33:13.821878+0000","last_fullsized":"2024-08-19T01:33:13.821878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:13:01.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.739173+0000","last_change":"2024-08-19T01:31:35.614962+0000","last_active":"2024-08-19T01:33:14.739173+0000","last_peered":"2024-08-19T01:33:14.739173+0000","last_clean":"2024-08-19T01:33:14.739173+0000","last_became_active":"2024-08-19T01:31:35.605904+0000","last_became_peered":"2024-08-19T01:31:35.605904+0000","last_unstale":"2024-08-19T01:33:14.739173+0000","last_undegraded":"2024-08-19T01:33:14.739173+0000","last_fullsized":"2024-08-19T01:33:14.739173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:26:36.102805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":3,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920351+0000","last_change":"2024-08-19T01:30:51.550448+0000","last_active":"2024-08-19T01:33:13.920351+0000","last_peered":"2024-08-19T01:33:13.920351+0000","last_clean":"2024-08-19T01:33:13.920351+0000","last_became_active":"2024-08-19T01:30:51.550074+0000","last_became_peered":"2024-08-19T01:30:51.550074+0000","last_unstale":"2024-08-19T01:33:13.920351+0000","last_undegraded":"2024-08-19T01:33:13.920351+0000","last_fullsized":"2024-08-19T01:33:13.920351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:33:13.016183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822063+0000","last_change":"2024-08-19T01:30:58.592077+0000","last_active":"2024-08-19T01:33:13.822063+0000","last_peered":"2024-08-19T01:33:13.822063+0000","last_clean":"2024-08-19T01:33:13.822063+0000","last_became_active":"2024-08-19T01:30:58.591651+0000","last_became_peered":"2024-08-19T01:30:58.591651+0000","last_unstale":"2024-08-19T01:33:13.822063+0000","last_undegraded":"2024-08-19T01:33:13.822063+0000","last_fullsized":"2024-08-19T01:33:13.822063+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:09:24.980954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009090+0000","last_change":"2024-08-19T01:30:51.552772+0000","last_active":"2024-08-19T01:33:14.009090+0000","last_peered":"2024-08-19T01:33:14.009090+0000","last_clean":"2024-08-19T01:33:14.009090+0000","last_became_active":"2024-08-19T01:30:51.550904+0000","last_became_peered":"2024-08-19T01:30:51.550904+0000","last_unstale":"2024-08-19T01:33:14.009090+0000","last_undegraded":"2024-08-19T01:33:14.009090+0000","last_fullsized":"2024-08-19T01:33:14.009090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:21:50.082587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821870+0000","last_change":"2024-08-19T01:30:51.559756+0000","last_active":"2024-08-19T01:33:13.821870+0000","last_peered":"2024-08-19T01:33:13.821870+0000","last_clean":"2024-08-19T01:33:13.821870+0000","last_became_active":"2024-08-19T01:30:51.559370+0000","last_became_peered":"2024-08-19T01:30:51.559370+0000","last_unstale":"2024-08-19T01:33:13.821870+0000","last_undegraded":"2024-08-19T01:33:13.821870+0000","last_fullsized":"2024-08-19T01:33:13.821870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:27:30.525346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821589+0000","last_change":"2024-08-19T01:30:51.556577+0000","last_active":"2024-08-19T01:33:13.821589+0000","last_peered":"2024-08-19T01:33:13.821589+0000","last_clean":"2024-08-19T01:33:13.821589+0000","last_became_active":"2024-08-19T01:30:51.555929+0000","last_became_peered":"2024-08-19T01:30:51.555929+0000","last_unstale":"2024-08-19T01:33:13.821589+0000","last_undegraded":"2024-08-19T01:33:13.821589+0000","last_fullsized":"2024-08-19T01:33:13.821589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:46:31.660752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010627+0000","last_change":"2024-08-19T01:30:51.558626+0000","last_active":"2024-08-19T01:33:14.010627+0000","last_peered":"2024-08-19T01:33:14.010627+0000","last_clean":"2024-08-19T01:33:14.010627+0000","last_became_active":"2024-08-19T01:30:51.558188+0000","last_became_peered":"2024-08-19T01:30:51.558188+0000","last_unstale":"2024-08-19T01:33:14.010627+0000","last_undegraded":"2024-08-19T01:33:14.010627+0000","last_fullsized":"2024-08-19T01:33:14.010627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:43:44.777643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010095+0000","last_change":"2024-08-19T01:30:51.552870+0000","last_active":"2024-08-19T01:33:14.010095+0000","last_peered":"2024-08-19T01:33:14.010095+0000","last_clean":"2024-08-19T01:33:14.010095+0000","last_became_active":"2024-08-19T01:30:51.552514+0000","last_became_peered":"2024-08-19T01:30:51.552514+0000","last_unstale":"2024-08-19T01:33:14.010095+0000","last_undegraded":"2024-08-19T01:33:14.010095+0000","last_fullsized":"2024-08-19T01:33:14.010095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:10:58.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009491+0000","last_change":"2024-08-19T01:30:51.554216+0000","last_active":"2024-08-19T01:33:14.009491+0000","last_peered":"2024-08-19T01:33:14.009491+0000","last_clean":"2024-08-19T01:33:14.009491+0000","last_became_active":"2024-08-19T01:30:51.552535+0000","last_became_peered":"2024-08-19T01:30:51.552535+0000","last_unstale":"2024-08-19T01:33:14.009491+0000","last_undegraded":"2024-08-19T01:33:14.009491+0000","last_fullsized":"2024-08-19T01:33:14.009491+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:26:52.971897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822254+0000","last_change":"2024-08-19T01:30:51.561703+0000","last_active":"2024-08-19T01:33:13.822254+0000","last_peered":"2024-08-19T01:33:13.822254+0000","last_clean":"2024-08-19T01:33:13.822254+0000","last_became_active":"2024-08-19T01:30:51.561385+0000","last_became_peered":"2024-08-19T01:30:51.561385+0000","last_unstale":"2024-08-19T01:33:13.822254+0000","last_undegraded":"2024-08-19T01:33:13.822254+0000","last_fullsized":"2024-08-19T01:33:13.822254+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:07:23.187023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820799+0000","last_change":"2024-08-19T01:30:51.564047+0000","last_active":"2024-08-19T01:33:13.820799+0000","last_peered":"2024-08-19T01:33:13.820799+0000","last_clean":"2024-08-19T01:33:13.820799+0000","last_became_active":"2024-08-19T01:30:51.563743+0000","last_became_peered":"2024-08-19T01:30:51.563743+0000","last_unstale":"2024-08-19T01:33:13.820799+0000","last_undegraded":"2024-08-19T01:33:13.820799+0000","last_fullsized":"2024-08-19T01:33:13.820799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:50:37.639496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929073+0000","last_change":"2024-08-19T01:30:51.557058+0000","last_active":"2024-08-19T01:33:13.929073+0000","last_peered":"2024-08-19T01:33:13.929073+0000","last_clean":"2024-08-19T01:33:13.929073+0000","last_became_active":"2024-08-19T01:30:51.554611+0000","last_became_peered":"2024-08-19T01:30:51.554611+0000","last_unstale":"2024-08-19T01:33:13.929073+0000","last_undegraded":"2024-08-19T01:33:13.929073+0000","last_fullsized":"2024-08-19T01:33:13.929073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:56:25.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820511+0000","last_change":"2024-08-19T01:30:51.564119+0000","last_active":"2024-08-19T01:33:13.820511+0000","last_peered":"2024-08-19T01:33:13.820511+0000","last_clean":"2024-08-19T01:33:13.820511+0000","last_became_active":"2024-08-19T01:30:51.563840+0000","last_became_peered":"2024-08-19T01:30:51.563840+0000","last_unstale":"2024-08-19T01:33:13.820511+0000","last_undegraded":"2024-08-19T01:33:13.820511+0000","last_fullsized":"2024-08-19T01:33:13.820511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:44:44.928760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009253+0000","last_change":"2024-08-19T01:30:51.561639+0000","last_active":"2024-08-19T01:33:14.009253+0000","last_peered":"2024-08-19T01:33:14.009253+0000","last_clean":"2024-08-19T01:33:14.009253+0000","last_became_active":"2024-08-19T01:30:51.561207+0000","last_became_peered":"2024-08-19T01:30:51.561207+0000","last_unstale":"2024-08-19T01:33:14.009253+0000","last_undegraded":"2024-08-19T01:33:14.009253+0000","last_fullsized":"2024-08-19T01:33:14.009253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:12:20.076628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011400+0000","last_change":"2024-08-19T01:30:51.556700+0000","last_active":"2024-08-19T01:33:14.011400+0000","last_peered":"2024-08-19T01:33:14.011400+0000","last_clean":"2024-08-19T01:33:14.011400+0000","last_became_active":"2024-08-19T01:30:51.556407+0000","last_became_peered":"2024-08-19T01:30:51.556407+0000","last_unstale":"2024-08-19T01:33:14.011400+0000","last_undegraded":"2024-08-19T01:33:14.011400+0000","last_fullsized":"2024-08-19T01:33:14.011400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:03:16.664253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009274+0000","last_change":"2024-08-19T01:30:51.553764+0000","last_active":"2024-08-19T01:33:14.009274+0000","last_peered":"2024-08-19T01:33:14.009274+0000","last_clean":"2024-08-19T01:33:14.009274+0000","last_became_active":"2024-08-19T01:30:51.553442+0000","last_became_peered":"2024-08-19T01:30:51.553442+0000","last_unstale":"2024-08-19T01:33:14.009274+0000","last_undegraded":"2024-08-19T01:33:14.009274+0000","last_fullsized":"2024-08-19T01:33:14.009274+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:36:44.476979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822255+0000","last_change":"2024-08-19T01:30:51.565528+0000","last_active":"2024-08-19T01:33:13.822255+0000","last_peered":"2024-08-19T01:33:13.822255+0000","last_clean":"2024-08-19T01:33:13.822255+0000","last_became_active":"2024-08-19T01:30:51.564649+0000","last_became_peered":"2024-08-19T01:30:51.564649+0000","last_unstale":"2024-08-19T01:33:13.822255+0000","last_undegraded":"2024-08-19T01:33:13.822255+0000","last_fullsized":"2024-08-19T01:33:13.822255+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:04:12.868496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.917383+0000","last_change":"2024-08-19T01:30:51.553425+0000","last_active":"2024-08-19T01:33:13.917383+0000","last_peered":"2024-08-19T01:33:13.917383+0000","last_clean":"2024-08-19T01:33:13.917383+0000","last_became_active":"2024-08-19T01:30:51.553123+0000","last_became_peered":"2024-08-19T01:30:51.553123+0000","last_unstale":"2024-08-19T01:33:13.917383+0000","last_undegraded":"2024-08-19T01:33:13.917383+0000","last_fullsized":"2024-08-19T01:33:13.917383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:07.969228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009636+0000","last_change":"2024-08-19T01:30:51.556820+0000","last_active":"2024-08-19T01:33:14.009636+0000","last_peered":"2024-08-19T01:33:14.009636+0000","last_clean":"2024-08-19T01:33:14.009636+0000","last_became_active":"2024-08-19T01:30:51.556532+0000","last_became_peered":"2024-08-19T01:30:51.556532+0000","last_unstale":"2024-08-19T01:33:14.009636+0000","last_undegraded":"2024-08-19T01:33:14.009636+0000","last_fullsized":"2024-08-19T01:33:14.009636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:40:37.053193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.008986+0000","last_change":"2024-08-19T01:30:51.561610+0000","last_active":"2024-08-19T01:33:14.008986+0000","last_peered":"2024-08-19T01:33:14.008986+0000","last_clean":"2024-08-19T01:33:14.008986+0000","last_became_active":"2024-08-19T01:30:51.561270+0000","last_became_peered":"2024-08-19T01:30:51.561270+0000","last_unstale":"2024-08-19T01:33:14.008986+0000","last_undegraded":"2024-08-19T01:33:14.008986+0000","last_fullsized":"2024-08-19T01:33:14.008986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:15:15.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010003+0000","last_change":"2024-08-19T01:30:51.558128+0000","last_active":"2024-08-19T01:33:14.010003+0000","last_peered":"2024-08-19T01:33:14.010003+0000","last_clean":"2024-08-19T01:33:14.010003+0000","last_became_active":"2024-08-19T01:30:51.557785+0000","last_became_peered":"2024-08-19T01:30:51.557785+0000","last_unstale":"2024-08-19T01:33:14.010003+0000","last_undegraded":"2024-08-19T01:33:14.010003+0000","last_fullsized":"2024-08-19T01:33:14.010003+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:14.849212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010225+0000","last_change":"2024-08-19T01:30:51.553917+0000","last_active":"2024-08-19T01:33:14.010225+0000","last_peered":"2024-08-19T01:33:14.010225+0000","last_clean":"2024-08-19T01:33:14.010225+0000","last_became_active":"2024-08-19T01:30:51.553596+0000","last_became_peered":"2024-08-19T01:30:51.553596+0000","last_unstale":"2024-08-19T01:33:14.010225+0000","last_undegraded":"2024-08-19T01:33:14.010225+0000","last_fullsized":"2024-08-19T01:33:14.010225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:03:36.001486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011300+0000","last_change":"2024-08-19T01:30:51.555201+0000","last_active":"2024-08-19T01:33:14.011300+0000","last_peered":"2024-08-19T01:33:14.011300+0000","last_clean":"2024-08-19T01:33:14.011300+0000","last_became_active":"2024-08-19T01:30:51.554160+0000","last_became_peered":"2024-08-19T01:30:51.554160+0000","last_unstale":"2024-08-19T01:33:14.011300+0000","last_undegraded":"2024-08-19T01:33:14.011300+0000","last_fullsized":"2024-08-19T01:33:14.011300+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:58:20.586931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821883+0000","last_change":"2024-08-19T01:30:51.560517+0000","last_active":"2024-08-19T01:33:13.821883+0000","last_peered":"2024-08-19T01:33:13.821883+0000","last_clean":"2024-08-19T01:33:13.821883+0000","last_became_active":"2024-08-19T01:30:51.557454+0000","last_became_peered":"2024-08-19T01:30:51.557454+0000","last_unstale":"2024-08-19T01:33:13.821883+0000","last_undegraded":"2024-08-19T01:33:13.821883+0000","last_fullsized":"2024-08-19T01:33:13.821883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:41:08.192165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010127+0000","last_change":"2024-08-19T01:30:51.565961+0000","last_active":"2024-08-19T01:33:14.010127+0000","last_peered":"2024-08-19T01:33:14.010127+0000","last_clean":"2024-08-19T01:33:14.010127+0000","last_became_active":"2024-08-19T01:30:51.565491+0000","last_became_peered":"2024-08-19T01:30:51.565491+0000","last_unstale":"2024-08-19T01:33:14.010127+0000","last_undegraded":"2024-08-19T01:33:14.010127+0000","last_fullsized":"2024-08-19T01:33:14.010127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:53:06.263411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932774+0000","last_change":"2024-08-19T01:30:51.551297+0000","last_active":"2024-08-19T01:33:13.932774+0000","last_peered":"2024-08-19T01:33:13.932774+0000","last_clean":"2024-08-19T01:33:13.932774+0000","last_became_active":"2024-08-19T01:30:51.550929+0000","last_became_peered":"2024-08-19T01:30:51.550929+0000","last_unstale":"2024-08-19T01:33:13.932774+0000","last_undegraded":"2024-08-19T01:33:13.932774+0000","last_fullsized":"2024-08-19T01:33:13.932774+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:07:03.625360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916372+0000","last_change":"2024-08-19T01:31:35.602883+0000","last_active":"2024-08-19T01:33:13.916372+0000","last_peered":"2024-08-19T01:33:13.916372+0000","last_clean":"2024-08-19T01:33:13.916372+0000","last_became_active":"2024-08-19T01:31:35.602571+0000","last_became_peered":"2024-08-19T01:31:35.602571+0000","last_unstale":"2024-08-19T01:33:13.916372+0000","last_undegraded":"2024-08-19T01:33:13.916372+0000","last_fullsized":"2024-08-19T01:33:13.916372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:08:13.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821015+0000","last_change":"2024-08-19T01:30:51.552548+0000","last_active":"2024-08-19T01:33:13.821015+0000","last_peered":"2024-08-19T01:33:13.821015+0000","last_clean":"2024-08-19T01:33:13.821015+0000","last_became_active":"2024-08-19T01:30:51.552264+0000","last_became_peered":"2024-08-19T01:30:51.552264+0000","last_unstale":"2024-08-19T01:33:13.821015+0000","last_undegraded":"2024-08-19T01:33:13.821015+0000","last_fullsized":"2024-08-19T01:33:13.821015+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:17:53.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009360+0000","last_change":"2024-08-19T01:30:51.558686+0000","last_active":"2024-08-19T01:33:14.009360+0000","last_peered":"2024-08-19T01:33:14.009360+0000","last_clean":"2024-08-19T01:33:14.009360+0000","last_became_active":"2024-08-19T01:30:51.558190+0000","last_became_peered":"2024-08-19T01:30:51.558190+0000","last_unstale":"2024-08-19T01:33:14.009360+0000","last_undegraded":"2024-08-19T01:33:14.009360+0000","last_fullsized":"2024-08-19T01:33:14.009360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:52:33.056948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932373+0000","last_change":"2024-08-19T01:30:51.548868+0000","last_active":"2024-08-19T01:33:13.932373+0000","last_peered":"2024-08-19T01:33:13.932373+0000","last_clean":"2024-08-19T01:33:13.932373+0000","last_became_active":"2024-08-19T01:30:51.548347+0000","last_became_peered":"2024-08-19T01:30:51.548347+0000","last_unstale":"2024-08-19T01:33:13.932373+0000","last_undegraded":"2024-08-19T01:33:13.932373+0000","last_fullsized":"2024-08-19T01:33:13.932373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:39:22.049290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920367+0000","last_change":"2024-08-19T01:30:51.554054+0000","last_active":"2024-08-19T01:33:13.920367+0000","last_peered":"2024-08-19T01:33:13.920367+0000","last_clean":"2024-08-19T01:33:13.920367+0000","last_became_active":"2024-08-19T01:30:51.553731+0000","last_became_peered":"2024-08-19T01:30:51.553731+0000","last_unstale":"2024-08-19T01:33:13.920367+0000","last_undegraded":"2024-08-19T01:33:13.920367+0000","last_fullsized":"2024-08-19T01:33:13.920367+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:25:17.939024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821646+0000","last_change":"2024-08-19T01:30:51.558662+0000","last_active":"2024-08-19T01:33:13.821646+0000","last_peered":"2024-08-19T01:33:13.821646+0000","last_clean":"2024-08-19T01:33:13.821646+0000","last_became_active":"2024-08-19T01:30:51.558131+0000","last_became_peered":"2024-08-19T01:30:51.558131+0000","last_unstale":"2024-08-19T01:33:13.821646+0000","last_undegraded":"2024-08-19T01:33:13.821646+0000","last_fullsized":"2024-08-19T01:33:13.821646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:31:25.054005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011016+0000","last_change":"2024-08-19T01:30:51.565170+0000","last_active":"2024-08-19T01:33:14.011016+0000","last_peered":"2024-08-19T01:33:14.011016+0000","last_clean":"2024-08-19T01:33:14.011016+0000","last_became_active":"2024-08-19T01:30:51.564765+0000","last_became_peered":"2024-08-19T01:30:51.564765+0000","last_unstale":"2024-08-19T01:33:14.011016+0000","last_undegraded":"2024-08-19T01:33:14.011016+0000","last_fullsized":"2024-08-19T01:33:14.011016+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:00:48.668865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011370+0000","last_change":"2024-08-19T01:30:51.555300+0000","last_active":"2024-08-19T01:33:14.011370+0000","last_peered":"2024-08-19T01:33:14.011370+0000","last_clean":"2024-08-19T01:33:14.011370+0000","last_became_active":"2024-08-19T01:30:51.554977+0000","last_became_peered":"2024-08-19T01:30:51.554977+0000","last_unstale":"2024-08-19T01:33:14.011370+0000","last_undegraded":"2024-08-19T01:33:14.011370+0000","last_fullsized":"2024-08-19T01:33:14.011370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:18:58.961783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009590+0000","last_change":"2024-08-19T01:30:51.557906+0000","last_active":"2024-08-19T01:33:14.009590+0000","last_peered":"2024-08-19T01:33:14.009590+0000","last_clean":"2024-08-19T01:33:14.009590+0000","last_became_active":"2024-08-19T01:30:51.557598+0000","last_became_peered":"2024-08-19T01:30:51.557598+0000","last_unstale":"2024-08-19T01:33:14.009590+0000","last_undegraded":"2024-08-19T01:33:14.009590+0000","last_fullsized":"2024-08-19T01:33:14.009590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:33:12.785910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599828,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":732,"num_read_kb":477,"num_write":298,"num_write_kb":1445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4463,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1799484,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107877,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481833,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":542338,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743470,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32364,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712788,"statfs":{"total":95995035648,"available":95961894912,"internally_reserved":0,"allocated":1216512,"data_stored":1335393,"data_compressed":74600,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906995,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":935790,"data_compressed":51631,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168630,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":935710,"data_compressed":51631,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430268,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":1140634,"data_compressed":68444,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593794,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":537750,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822726,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1266511,"data_compressed":74600,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712872,"statfs":{"total":95995035648,"available":95961980928,"internally_reserved":0,"allocated":1126400,"data_stored":612491,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215181,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":537944,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":528464,"data_compressed":29513,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":608373,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":529214,"data_compressed":29513,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":70713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134387,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":203269,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T01:33:20.895 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-19T01:33:20.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T01:33:21.150 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:21.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:20 smithi123 ceph-mon[21829]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-08-19T01:33:21.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:20 smithi123 ceph-mon[21829]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:21.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:20 smithi161 ceph-mon[21757]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-08-19T01:33:21.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:20 smithi161 ceph-mon[21757]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:20 smithi064 ceph-mon[18515]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-08-19T01:33:21.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:20 smithi064 ceph-mon[18515]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:21.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 -- 172.21.15.64:0/2566822507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4102c20 msgr2=0x7fb6a410d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 --2- 172.21.15.64:0/2566822507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4102c20 0x7fb6a410d770 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009ca0 tx=0x7fb69c02a950 comp rx=0 tx=0).stop 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 -- 172.21.15.64:0/2566822507 shutdown_connections 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 --2- 172.21.15.64:0/2566822507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4102c20 0x7fb6a410d770 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 --2- 172.21.15.64:0/2566822507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6a41022a0 0x7fb6a41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 --2- 172.21.15.64:0/2566822507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a4100b50 0x7fb6a4100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 -- 172.21.15.64:0/2566822507 >> 172.21.15.64:0/2566822507 conn(0x7fb6a40fc500 msgr2=0x7fb6a40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 -- 172.21.15.64:0/2566822507 shutdown_connections 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.652+0000 7fb6aa851640 1 -- 172.21.15.64:0/2566822507 wait complete. 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 Processor -- start 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 -- start start 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4100b50 0x7fb6a4198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6a41022a0 0x7fb6a4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:21.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a4102c20 0x7fb6a419caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:21.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a4111e20 con 0x7fb6a4100b50 2024-08-19T01:33:21.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a4111ca0 con 0x7fb6a4102c20 2024-08-19T01:33:21.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6aa851640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a4111fa0 con 0x7fb6a41022a0 2024-08-19T01:33:21.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6a3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4100b50 0x7fb6a4198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:21.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6a3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4100b50 0x7fb6a4198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50878/0 (socket says 172.21.15.64:50878) 2024-08-19T01:33:21.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.653+0000 7fb6a3fff640 1 -- 172.21.15.64:0/1404660501 learned_addr learned my addr 172.21.15.64:0/1404660501 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb69bfff640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6a41022a0 0x7fb6a4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a8dee640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a4102c20 0x7fb6a419caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a3fff640 1 -- 172.21.15.64:0/1404660501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6a41022a0 msgr2=0x7fb6a4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a3fff640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6a41022a0 0x7fb6a4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a3fff640 1 -- 172.21.15.64:0/1404660501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a4102c20 msgr2=0x7fb6a419caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a3fff640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a4102c20 0x7fb6a419caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a3fff640 1 -- 172.21.15.64:0/1404660501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a419d1d0 con 0x7fb6a4100b50 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb69bfff640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6a41022a0 0x7fb6a4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a3fff640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4100b50 0x7fb6a4198220 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb69400c890 tx=0x7fb69400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a1ffb640 1 -- 172.21.15.64:0/1404660501 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694004220 con 0x7fb6a4100b50 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a419d4c0 con 0x7fb6a4100b50 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a1ffb640 1 -- 172.21.15.64:0/1404660501 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6940043c0 con 0x7fb6a4100b50 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6a1ffb640 1 -- 172.21.15.64:0/1404660501 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6940049e0 con 0x7fb6a4100b50 2024-08-19T01:33:21.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.654+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a41a4da0 con 0x7fb6a4100b50 2024-08-19T01:33:21.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.655+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a4100f30 con 0x7fb6a4100b50 2024-08-19T01:33:21.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.656+0000 7fb6a1ffb640 1 -- 172.21.15.64:0/1404660501 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fb6940079b0 con 0x7fb6a4100b50 2024-08-19T01:33:21.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.657+0000 7fb6a1ffb640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb6900a2e10 0x7fb6900a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:21.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.657+0000 7fb69bfff640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb6900a2e10 0x7fb6900a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:21.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.657+0000 7fb6a1ffb640 1 -- 172.21.15.64:0/1404660501 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb6940cfbf0 con 0x7fb6a4100b50 2024-08-19T01:33:21.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.657+0000 7fb69bfff640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb6900a2e10 0x7fb6900a52d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb68c006fd0 tx=0x7fb68c008040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:21.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.661+0000 7fb6a1ffb640 1 -- 172.21.15.64:0/1404660501 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb694014030 con 0x7fb6a4100b50 2024-08-19T01:33:21.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.833+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6a4106c70 con 0x7fb6900a2e10 2024-08-19T01:33:21.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.841+0000 7fb6a1ffb640 1 -- 172.21.15.64:0/1404660501 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339711 (secure 0 0 0) 0x7fb6a4106c70 con 0x7fb6900a2e10 2024-08-19T01:33:21.846 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:21.847 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-08-19T01:33:21.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.844+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb6900a2e10 msgr2=0x7fb6900a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:21.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.844+0000 7fb6aa851640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb6900a2e10 0x7fb6900a52d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb68c006fd0 tx=0x7fb68c008040 comp rx=0 tx=0).stop 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.844+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4100b50 msgr2=0x7fb6a4198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.844+0000 7fb6aa851640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4100b50 0x7fb6a4198220 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb69400c890 tx=0x7fb69400cd50 comp rx=0 tx=0).stop 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 shutdown_connections 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb6900a2e10 0x7fb6900a52d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a4102c20 0x7fb6a419caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6a41022a0 0x7fb6a4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 --2- 172.21.15.64:0/1404660501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6a4100b50 0x7fb6a4198220 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 >> 172.21.15.64:0/1404660501 conn(0x7fb6a40fc500 msgr2=0x7fb6a4106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 shutdown_connections 2024-08-19T01:33:21.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:21.845+0000 7fb6aa851640 1 -- 172.21.15.64:0/1404660501 wait complete. 2024-08-19T01:33:22.553 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-08-19T01:33:21.833750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1390362,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1250,"num_read_kb":1052,"num_write":634,"num_write_kb":5306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1185316,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":967,"ondisk_log_size":967,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366272,"kb_used_data":12920,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575552,"statfs":{"total":1151940427776,"available":1151565365248,"internally_reserved":0,"allocated":13230080,"data_stored":10655072,"data_compressed":601170,"data_compressed_allocated":811008,"data_compressed_original":4952064,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":199932,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":387,"num_read_kb":461,"num_write":153,"num_write_kb":921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.992110"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916960+0000","last_change":"2024-08-19T01:30:49.539512+0000","last_active":"2024-08-19T01:33:13.916960+0000","last_peered":"2024-08-19T01:33:13.916960+0000","last_clean":"2024-08-19T01:33:13.916960+0000","last_became_active":"2024-08-19T01:30:49.539163+0000","last_became_peered":"2024-08-19T01:30:49.539163+0000","last_unstale":"2024-08-19T01:33:13.916960+0000","last_undegraded":"2024-08-19T01:33:13.916960+0000","last_fullsized":"2024-08-19T01:33:13.916960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:23.874916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821250+0000","last_change":"2024-08-19T01:30:51.561135+0000","last_active":"2024-08-19T01:33:13.821250+0000","last_peered":"2024-08-19T01:33:13.821250+0000","last_clean":"2024-08-19T01:33:13.821250+0000","last_became_active":"2024-08-19T01:30:51.560822+0000","last_became_peered":"2024-08-19T01:30:51.560822+0000","last_unstale":"2024-08-19T01:33:13.821250+0000","last_undegraded":"2024-08-19T01:33:13.821250+0000","last_fullsized":"2024-08-19T01:33:13.821250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:08:35.918895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821530+0000","last_change":"2024-08-19T01:30:58.595727+0000","last_active":"2024-08-19T01:33:13.821530+0000","last_peered":"2024-08-19T01:33:13.821530+0000","last_clean":"2024-08-19T01:33:13.821530+0000","last_became_active":"2024-08-19T01:30:58.595286+0000","last_became_peered":"2024-08-19T01:30:58.595286+0000","last_unstale":"2024-08-19T01:33:13.821530+0000","last_undegraded":"2024-08-19T01:33:13.821530+0000","last_fullsized":"2024-08-19T01:33:13.821530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:25:02.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821435+0000","last_change":"2024-08-19T01:30:49.575693+0000","last_active":"2024-08-19T01:33:13.821435+0000","last_peered":"2024-08-19T01:33:13.821435+0000","last_clean":"2024-08-19T01:33:13.821435+0000","last_became_active":"2024-08-19T01:30:49.575331+0000","last_became_peered":"2024-08-19T01:30:49.575331+0000","last_unstale":"2024-08-19T01:33:13.821435+0000","last_undegraded":"2024-08-19T01:33:13.821435+0000","last_fullsized":"2024-08-19T01:33:13.821435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:31:50.773160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011817+0000","last_change":"2024-08-19T01:30:51.562446+0000","last_active":"2024-08-19T01:33:14.011817+0000","last_peered":"2024-08-19T01:33:14.011817+0000","last_clean":"2024-08-19T01:33:14.011817+0000","last_became_active":"2024-08-19T01:30:51.561475+0000","last_became_peered":"2024-08-19T01:30:51.561475+0000","last_unstale":"2024-08-19T01:33:14.011817+0000","last_undegraded":"2024-08-19T01:33:14.011817+0000","last_fullsized":"2024-08-19T01:33:14.011817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:51:41.531247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821367+0000","last_change":"2024-08-19T01:30:58.591371+0000","last_active":"2024-08-19T01:33:13.821367+0000","last_peered":"2024-08-19T01:33:13.821367+0000","last_clean":"2024-08-19T01:33:13.821367+0000","last_became_active":"2024-08-19T01:30:58.590936+0000","last_became_peered":"2024-08-19T01:30:58.590936+0000","last_unstale":"2024-08-19T01:33:13.821367+0000","last_undegraded":"2024-08-19T01:33:13.821367+0000","last_fullsized":"2024-08-19T01:33:13.821367+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:07:33.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009287+0000","last_change":"2024-08-19T01:31:35.613145+0000","last_active":"2024-08-19T01:33:14.009287+0000","last_peered":"2024-08-19T01:33:14.009287+0000","last_clean":"2024-08-19T01:33:14.009287+0000","last_became_active":"2024-08-19T01:31:35.605087+0000","last_became_peered":"2024-08-19T01:31:35.605087+0000","last_unstale":"2024-08-19T01:33:14.009287+0000","last_undegraded":"2024-08-19T01:33:14.009287+0000","last_fullsized":"2024-08-19T01:33:14.009287+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:25:59.109707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009849+0000","last_change":"2024-08-19T01:30:51.555256+0000","last_active":"2024-08-19T01:33:14.009849+0000","last_peered":"2024-08-19T01:33:14.009849+0000","last_clean":"2024-08-19T01:33:14.009849+0000","last_became_active":"2024-08-19T01:30:51.554030+0000","last_became_peered":"2024-08-19T01:30:51.554030+0000","last_unstale":"2024-08-19T01:33:14.009849+0000","last_undegraded":"2024-08-19T01:33:14.009849+0000","last_fullsized":"2024-08-19T01:33:14.009849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:12:04.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821647+0000","last_change":"2024-08-19T01:30:58.589362+0000","last_active":"2024-08-19T01:33:13.821647+0000","last_peered":"2024-08-19T01:33:13.821647+0000","last_clean":"2024-08-19T01:33:13.821647+0000","last_became_active":"2024-08-19T01:30:58.588937+0000","last_became_peered":"2024-08-19T01:30:58.588937+0000","last_unstale":"2024-08-19T01:33:13.821647+0000","last_undegraded":"2024-08-19T01:33:13.821647+0000","last_fullsized":"2024-08-19T01:33:13.821647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:31.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920634+0000","last_change":"2024-08-19T01:30:49.538597+0000","last_active":"2024-08-19T01:33:13.920634+0000","last_peered":"2024-08-19T01:33:13.920634+0000","last_clean":"2024-08-19T01:33:13.920634+0000","last_became_active":"2024-08-19T01:30:49.538228+0000","last_became_peered":"2024-08-19T01:30:49.538228+0000","last_unstale":"2024-08-19T01:33:13.920634+0000","last_undegraded":"2024-08-19T01:33:13.920634+0000","last_fullsized":"2024-08-19T01:33:13.920634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:17:31.350962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929475+0000","last_change":"2024-08-19T01:30:51.562829+0000","last_active":"2024-08-19T01:33:13.929475+0000","last_peered":"2024-08-19T01:33:13.929475+0000","last_clean":"2024-08-19T01:33:13.929475+0000","last_became_active":"2024-08-19T01:30:51.562144+0000","last_became_peered":"2024-08-19T01:30:51.562144+0000","last_unstale":"2024-08-19T01:33:13.929475+0000","last_undegraded":"2024-08-19T01:33:13.929475+0000","last_fullsized":"2024-08-19T01:33:13.929475+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:17:09.184222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821477+0000","last_change":"2024-08-19T01:30:58.593178+0000","last_active":"2024-08-19T01:33:13.821477+0000","last_peered":"2024-08-19T01:33:13.821477+0000","last_clean":"2024-08-19T01:33:13.821477+0000","last_became_active":"2024-08-19T01:30:58.591247+0000","last_became_peered":"2024-08-19T01:30:58.591247+0000","last_unstale":"2024-08-19T01:33:13.821477+0000","last_undegraded":"2024-08-19T01:33:13.821477+0000","last_fullsized":"2024-08-19T01:33:13.821477+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:20:40.796135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009302+0000","last_change":"2024-08-19T01:30:51.562307+0000","last_active":"2024-08-19T01:33:14.009302+0000","last_peered":"2024-08-19T01:33:14.009302+0000","last_clean":"2024-08-19T01:33:14.009302+0000","last_became_active":"2024-08-19T01:30:51.561957+0000","last_became_peered":"2024-08-19T01:30:51.561957+0000","last_unstale":"2024-08-19T01:33:14.009302+0000","last_undegraded":"2024-08-19T01:33:14.009302+0000","last_fullsized":"2024-08-19T01:33:14.009302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:22.902317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009176+0000","last_change":"2024-08-19T01:31:35.609612+0000","last_active":"2024-08-19T01:33:14.009176+0000","last_peered":"2024-08-19T01:33:14.009176+0000","last_clean":"2024-08-19T01:33:14.009176+0000","last_became_active":"2024-08-19T01:31:35.604969+0000","last_became_peered":"2024-08-19T01:31:35.604969+0000","last_unstale":"2024-08-19T01:33:14.009176+0000","last_undegraded":"2024-08-19T01:33:14.009176+0000","last_fullsized":"2024-08-19T01:33:14.009176+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:59:38.089119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821743+0000","last_change":"2024-08-19T01:30:58.588984+0000","last_active":"2024-08-19T01:33:13.821743+0000","last_peered":"2024-08-19T01:33:13.821743+0000","last_clean":"2024-08-19T01:33:13.821743+0000","last_became_active":"2024-08-19T01:30:58.588504+0000","last_became_peered":"2024-08-19T01:30:58.588504+0000","last_unstale":"2024-08-19T01:33:13.821743+0000","last_undegraded":"2024-08-19T01:33:13.821743+0000","last_fullsized":"2024-08-19T01:33:13.821743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:29:14.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820715+0000","last_change":"2024-08-19T01:30:51.553293+0000","last_active":"2024-08-19T01:33:13.820715+0000","last_peered":"2024-08-19T01:33:13.820715+0000","last_clean":"2024-08-19T01:33:13.820715+0000","last_became_active":"2024-08-19T01:30:51.552990+0000","last_became_peered":"2024-08-19T01:30:51.552990+0000","last_unstale":"2024-08-19T01:33:13.820715+0000","last_undegraded":"2024-08-19T01:33:13.820715+0000","last_fullsized":"2024-08-19T01:33:13.820715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:28:33.193014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821634+0000","last_change":"2024-08-19T01:30:49.537742+0000","last_active":"2024-08-19T01:33:13.821634+0000","last_peered":"2024-08-19T01:33:13.821634+0000","last_clean":"2024-08-19T01:33:13.821634+0000","last_became_active":"2024-08-19T01:30:49.537153+0000","last_became_peered":"2024-08-19T01:30:49.537153+0000","last_unstale":"2024-08-19T01:33:13.821634+0000","last_undegraded":"2024-08-19T01:33:13.821634+0000","last_fullsized":"2024-08-19T01:33:13.821634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:25:03.650900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821745+0000","last_change":"2024-08-19T01:32:35.668059+0000","last_active":"2024-08-19T01:33:13.821745+0000","last_peered":"2024-08-19T01:33:13.821745+0000","last_clean":"2024-08-19T01:33:13.821745+0000","last_became_active":"2024-08-19T01:32:35.663768+0000","last_became_peered":"2024-08-19T01:32:35.663768+0000","last_unstale":"2024-08-19T01:33:13.821745+0000","last_undegraded":"2024-08-19T01:33:13.821745+0000","last_fullsized":"2024-08-19T01:33:13.821745+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:19:38.669783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009175+0000","last_change":"2024-08-19T01:30:51.552126+0000","last_active":"2024-08-19T01:33:14.009175+0000","last_peered":"2024-08-19T01:33:14.009175+0000","last_clean":"2024-08-19T01:33:14.009175+0000","last_became_active":"2024-08-19T01:30:51.551795+0000","last_became_peered":"2024-08-19T01:30:51.551795+0000","last_unstale":"2024-08-19T01:33:14.009175+0000","last_undegraded":"2024-08-19T01:33:14.009175+0000","last_fullsized":"2024-08-19T01:33:14.009175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:22:54.705359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820641+0000","last_change":"2024-08-19T01:30:49.536659+0000","last_active":"2024-08-19T01:33:13.820641+0000","last_peered":"2024-08-19T01:33:13.820641+0000","last_clean":"2024-08-19T01:33:13.820641+0000","last_became_active":"2024-08-19T01:30:49.536366+0000","last_became_peered":"2024-08-19T01:30:49.536366+0000","last_unstale":"2024-08-19T01:33:13.820641+0000","last_undegraded":"2024-08-19T01:33:13.820641+0000","last_fullsized":"2024-08-19T01:33:13.820641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:58:52.547461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916663+0000","last_change":"2024-08-19T01:30:58.593724+0000","last_active":"2024-08-19T01:33:13.916663+0000","last_peered":"2024-08-19T01:33:13.916663+0000","last_clean":"2024-08-19T01:33:13.916663+0000","last_became_active":"2024-08-19T01:30:58.593383+0000","last_became_peered":"2024-08-19T01:30:58.593383+0000","last_unstale":"2024-08-19T01:33:13.916663+0000","last_undegraded":"2024-08-19T01:33:13.916663+0000","last_fullsized":"2024-08-19T01:33:13.916663+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:54:26.424139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920243+0000","last_change":"2024-08-19T01:30:51.557682+0000","last_active":"2024-08-19T01:33:13.920243+0000","last_peered":"2024-08-19T01:33:13.920243+0000","last_clean":"2024-08-19T01:33:13.920243+0000","last_became_active":"2024-08-19T01:30:51.557351+0000","last_became_peered":"2024-08-19T01:30:51.557351+0000","last_unstale":"2024-08-19T01:33:13.920243+0000","last_undegraded":"2024-08-19T01:33:13.920243+0000","last_fullsized":"2024-08-19T01:33:13.920243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:44:41.484955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009059+0000","last_change":"2024-08-19T01:30:49.575996+0000","last_active":"2024-08-19T01:33:14.009059+0000","last_peered":"2024-08-19T01:33:14.009059+0000","last_clean":"2024-08-19T01:33:14.009059+0000","last_became_active":"2024-08-19T01:30:49.575607+0000","last_became_peered":"2024-08-19T01:30:49.575607+0000","last_unstale":"2024-08-19T01:33:14.009059+0000","last_undegraded":"2024-08-19T01:33:14.009059+0000","last_fullsized":"2024-08-19T01:33:14.009059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:20:33.467100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821854+0000","last_change":"2024-08-19T01:30:58.592629+0000","last_active":"2024-08-19T01:33:13.821854+0000","last_peered":"2024-08-19T01:33:13.821854+0000","last_clean":"2024-08-19T01:33:13.821854+0000","last_became_active":"2024-08-19T01:30:58.592122+0000","last_became_peered":"2024-08-19T01:30:58.592122+0000","last_unstale":"2024-08-19T01:33:13.821854+0000","last_undegraded":"2024-08-19T01:33:13.821854+0000","last_fullsized":"2024-08-19T01:33:13.821854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:19:09.846829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011482+0000","last_change":"2024-08-19T01:31:35.746416+0000","last_active":"2024-08-19T01:33:14.011482+0000","last_peered":"2024-08-19T01:33:14.011482+0000","last_clean":"2024-08-19T01:33:14.011482+0000","last_became_active":"2024-08-19T01:31:35.746110+0000","last_became_peered":"2024-08-19T01:31:35.746110+0000","last_unstale":"2024-08-19T01:33:14.011482+0000","last_undegraded":"2024-08-19T01:33:14.011482+0000","last_fullsized":"2024-08-19T01:33:14.011482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:29:29.789785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821949+0000","last_change":"2024-08-19T01:30:49.711366+0000","last_active":"2024-08-19T01:33:13.821949+0000","last_peered":"2024-08-19T01:33:13.821949+0000","last_clean":"2024-08-19T01:33:13.821949+0000","last_became_active":"2024-08-19T01:30:49.711078+0000","last_became_peered":"2024-08-19T01:30:49.711078+0000","last_unstale":"2024-08-19T01:33:13.821949+0000","last_undegraded":"2024-08-19T01:33:13.821949+0000","last_fullsized":"2024-08-19T01:33:13.821949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:11:16.767867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821530+0000","last_change":"2024-08-19T01:30:58.591845+0000","last_active":"2024-08-19T01:33:13.821530+0000","last_peered":"2024-08-19T01:33:13.821530+0000","last_clean":"2024-08-19T01:33:13.821530+0000","last_became_active":"2024-08-19T01:30:58.591453+0000","last_became_peered":"2024-08-19T01:30:58.591453+0000","last_unstale":"2024-08-19T01:33:13.821530+0000","last_undegraded":"2024-08-19T01:33:13.821530+0000","last_fullsized":"2024-08-19T01:33:13.821530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:37:09.521514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820925+0000","last_change":"2024-08-19T01:30:51.563156+0000","last_active":"2024-08-19T01:33:13.820925+0000","last_peered":"2024-08-19T01:33:13.820925+0000","last_clean":"2024-08-19T01:33:13.820925+0000","last_became_active":"2024-08-19T01:30:51.562860+0000","last_became_peered":"2024-08-19T01:30:51.562860+0000","last_unstale":"2024-08-19T01:33:13.820925+0000","last_undegraded":"2024-08-19T01:33:13.820925+0000","last_fullsized":"2024-08-19T01:33:13.820925+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:54:11.768833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'102","reported_seq":235,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:17.436431+0000","last_change":"2024-08-19T01:30:49.536648+0000","last_active":"2024-08-19T01:33:17.436431+0000","last_peered":"2024-08-19T01:33:17.436431+0000","last_clean":"2024-08-19T01:33:17.436431+0000","last_became_active":"2024-08-19T01:30:49.536080+0000","last_became_peered":"2024-08-19T01:30:49.536080+0000","last_unstale":"2024-08-19T01:33:17.436431+0000","last_undegraded":"2024-08-19T01:33:17.436431+0000","last_fullsized":"2024-08-19T01:33:17.436431+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:03:46.804252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":188,"num_write_kb":1288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820301+0000","last_change":"2024-08-19T01:30:58.592253+0000","last_active":"2024-08-19T01:33:13.820301+0000","last_peered":"2024-08-19T01:33:13.820301+0000","last_clean":"2024-08-19T01:33:13.820301+0000","last_became_active":"2024-08-19T01:30:58.591911+0000","last_became_peered":"2024-08-19T01:30:58.591911+0000","last_unstale":"2024-08-19T01:33:13.820301+0000","last_undegraded":"2024-08-19T01:33:13.820301+0000","last_fullsized":"2024-08-19T01:33:13.820301+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:02:27.112597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010224+0000","last_change":"2024-08-19T01:30:51.547607+0000","last_active":"2024-08-19T01:33:14.010224+0000","last_peered":"2024-08-19T01:33:14.010224+0000","last_clean":"2024-08-19T01:33:14.010224+0000","last_became_active":"2024-08-19T01:30:51.547316+0000","last_became_peered":"2024-08-19T01:30:51.547316+0000","last_unstale":"2024-08-19T01:33:14.010224+0000","last_undegraded":"2024-08-19T01:33:14.010224+0000","last_fullsized":"2024-08-19T01:33:14.010224+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:48:58.300508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920697+0000","last_change":"2024-08-19T01:30:49.576942+0000","last_active":"2024-08-19T01:33:13.920697+0000","last_peered":"2024-08-19T01:33:13.920697+0000","last_clean":"2024-08-19T01:33:13.920697+0000","last_became_active":"2024-08-19T01:30:49.576528+0000","last_became_peered":"2024-08-19T01:30:49.576528+0000","last_unstale":"2024-08-19T01:33:13.920697+0000","last_undegraded":"2024-08-19T01:33:13.920697+0000","last_fullsized":"2024-08-19T01:33:13.920697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:23:53.259395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009281+0000","last_change":"2024-08-19T01:30:58.591577+0000","last_active":"2024-08-19T01:33:14.009281+0000","last_peered":"2024-08-19T01:33:14.009281+0000","last_clean":"2024-08-19T01:33:14.009281+0000","last_became_active":"2024-08-19T01:30:58.591181+0000","last_became_peered":"2024-08-19T01:30:58.591181+0000","last_unstale":"2024-08-19T01:33:14.009281+0000","last_undegraded":"2024-08-19T01:33:14.009281+0000","last_fullsized":"2024-08-19T01:33:14.009281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:02:59.682147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932531+0000","last_change":"2024-08-19T01:30:51.548556+0000","last_active":"2024-08-19T01:33:13.932531+0000","last_peered":"2024-08-19T01:33:13.932531+0000","last_clean":"2024-08-19T01:33:13.932531+0000","last_became_active":"2024-08-19T01:30:51.547838+0000","last_became_peered":"2024-08-19T01:30:51.547838+0000","last_unstale":"2024-08-19T01:33:13.932531+0000","last_undegraded":"2024-08-19T01:33:13.932531+0000","last_fullsized":"2024-08-19T01:33:13.932531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:08:49.671964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916085+0000","last_change":"2024-08-19T01:30:49.576799+0000","last_active":"2024-08-19T01:33:13.916085+0000","last_peered":"2024-08-19T01:33:13.916085+0000","last_clean":"2024-08-19T01:33:13.916085+0000","last_became_active":"2024-08-19T01:30:49.576385+0000","last_became_peered":"2024-08-19T01:30:49.576385+0000","last_unstale":"2024-08-19T01:33:13.916085+0000","last_undegraded":"2024-08-19T01:33:13.916085+0000","last_fullsized":"2024-08-19T01:33:13.916085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:34:18.287518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920261+0000","last_change":"2024-08-19T01:30:58.593703+0000","last_active":"2024-08-19T01:33:13.920261+0000","last_peered":"2024-08-19T01:33:13.920261+0000","last_clean":"2024-08-19T01:33:13.920261+0000","last_became_active":"2024-08-19T01:30:58.593303+0000","last_became_peered":"2024-08-19T01:30:58.593303+0000","last_unstale":"2024-08-19T01:33:13.920261+0000","last_undegraded":"2024-08-19T01:33:13.920261+0000","last_fullsized":"2024-08-19T01:33:13.920261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:12:43.359262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932455+0000","last_change":"2024-08-19T01:31:35.601173+0000","last_active":"2024-08-19T01:33:13.932455+0000","last_peered":"2024-08-19T01:33:13.932455+0000","last_clean":"2024-08-19T01:33:13.932455+0000","last_became_active":"2024-08-19T01:31:35.600742+0000","last_became_peered":"2024-08-19T01:31:35.600742+0000","last_unstale":"2024-08-19T01:33:13.932455+0000","last_undegraded":"2024-08-19T01:33:13.932455+0000","last_fullsized":"2024-08-19T01:33:13.932455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:36:48.977124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009550+0000","last_change":"2024-08-19T01:30:49.540070+0000","last_active":"2024-08-19T01:33:14.009550+0000","last_peered":"2024-08-19T01:33:14.009550+0000","last_clean":"2024-08-19T01:33:14.009550+0000","last_became_active":"2024-08-19T01:30:49.539706+0000","last_became_peered":"2024-08-19T01:30:49.539706+0000","last_unstale":"2024-08-19T01:33:14.009550+0000","last_undegraded":"2024-08-19T01:33:14.009550+0000","last_fullsized":"2024-08-19T01:33:14.009550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:14:15.813670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009939+0000","last_change":"2024-08-19T01:30:58.590766+0000","last_active":"2024-08-19T01:33:14.009939+0000","last_peered":"2024-08-19T01:33:14.009939+0000","last_clean":"2024-08-19T01:33:14.009939+0000","last_became_active":"2024-08-19T01:30:58.590415+0000","last_became_peered":"2024-08-19T01:30:58.590415+0000","last_unstale":"2024-08-19T01:33:14.009939+0000","last_undegraded":"2024-08-19T01:33:14.009939+0000","last_fullsized":"2024-08-19T01:33:14.009939+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:48:59.261086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914324+0000","last_change":"2024-08-19T01:31:35.602332+0000","last_active":"2024-08-19T01:33:13.914324+0000","last_peered":"2024-08-19T01:33:13.914324+0000","last_clean":"2024-08-19T01:33:13.914324+0000","last_became_active":"2024-08-19T01:31:35.602007+0000","last_became_peered":"2024-08-19T01:31:35.602007+0000","last_unstale":"2024-08-19T01:33:13.914324+0000","last_undegraded":"2024-08-19T01:33:13.914324+0000","last_fullsized":"2024-08-19T01:33:13.914324+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:17:47.694567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":153,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:18.743740+0000","last_change":"2024-08-19T01:30:49.575995+0000","last_active":"2024-08-19T01:33:18.743740+0000","last_peered":"2024-08-19T01:33:18.743740+0000","last_clean":"2024-08-19T01:33:18.743740+0000","last_became_active":"2024-08-19T01:30:49.575249+0000","last_became_peered":"2024-08-19T01:30:49.575249+0000","last_unstale":"2024-08-19T01:33:18.743740+0000","last_undegraded":"2024-08-19T01:33:18.743740+0000","last_fullsized":"2024-08-19T01:33:18.743740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:39:05.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011011+0000","last_change":"2024-08-19T01:32:35.725750+0000","last_active":"2024-08-19T01:33:14.011011+0000","last_peered":"2024-08-19T01:33:14.011011+0000","last_clean":"2024-08-19T01:33:14.011011+0000","last_became_active":"2024-08-19T01:32:35.725320+0000","last_became_peered":"2024-08-19T01:32:35.725320+0000","last_unstale":"2024-08-19T01:33:14.011011+0000","last_undegraded":"2024-08-19T01:33:14.011011+0000","last_fullsized":"2024-08-19T01:33:14.011011+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:41:10.492801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009295+0000","last_change":"2024-08-19T01:30:51.549158+0000","last_active":"2024-08-19T01:33:14.009295+0000","last_peered":"2024-08-19T01:33:14.009295+0000","last_clean":"2024-08-19T01:33:14.009295+0000","last_became_active":"2024-08-19T01:30:51.548854+0000","last_became_peered":"2024-08-19T01:30:51.548854+0000","last_unstale":"2024-08-19T01:33:14.009295+0000","last_undegraded":"2024-08-19T01:33:14.009295+0000","last_fullsized":"2024-08-19T01:33:14.009295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:02:35.661758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009708+0000","last_change":"2024-08-19T01:30:49.538282+0000","last_active":"2024-08-19T01:33:14.009708+0000","last_peered":"2024-08-19T01:33:14.009708+0000","last_clean":"2024-08-19T01:33:14.009708+0000","last_became_active":"2024-08-19T01:30:49.537918+0000","last_became_peered":"2024-08-19T01:30:49.537918+0000","last_unstale":"2024-08-19T01:33:14.009708+0000","last_undegraded":"2024-08-19T01:33:14.009708+0000","last_fullsized":"2024-08-19T01:33:14.009708+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:27.886958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929499+0000","last_change":"2024-08-19T01:30:58.591153+0000","last_active":"2024-08-19T01:33:13.929499+0000","last_peered":"2024-08-19T01:33:13.929499+0000","last_clean":"2024-08-19T01:33:13.929499+0000","last_became_active":"2024-08-19T01:30:58.590738+0000","last_became_peered":"2024-08-19T01:30:58.590738+0000","last_unstale":"2024-08-19T01:33:13.929499+0000","last_undegraded":"2024-08-19T01:33:13.929499+0000","last_fullsized":"2024-08-19T01:33:13.929499+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:29.073825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820667+0000","last_change":"2024-08-19T01:30:51.554878+0000","last_active":"2024-08-19T01:33:13.820667+0000","last_peered":"2024-08-19T01:33:13.820667+0000","last_clean":"2024-08-19T01:33:13.820667+0000","last_became_active":"2024-08-19T01:30:51.553743+0000","last_became_peered":"2024-08-19T01:30:51.553743+0000","last_unstale":"2024-08-19T01:33:13.820667+0000","last_undegraded":"2024-08-19T01:33:13.820667+0000","last_fullsized":"2024-08-19T01:33:13.820667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:08:41.932641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822048+0000","last_change":"2024-08-19T01:30:49.542043+0000","last_active":"2024-08-19T01:33:13.822048+0000","last_peered":"2024-08-19T01:33:13.822048+0000","last_clean":"2024-08-19T01:33:13.822048+0000","last_became_active":"2024-08-19T01:30:49.541551+0000","last_became_peered":"2024-08-19T01:30:49.541551+0000","last_unstale":"2024-08-19T01:33:13.822048+0000","last_undegraded":"2024-08-19T01:33:13.822048+0000","last_fullsized":"2024-08-19T01:33:13.822048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:46:14.781516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":415,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:15.933182+0000","last_change":"2024-08-19T01:29:41.574391+0000","last_active":"2024-08-19T01:33:15.933182+0000","last_peered":"2024-08-19T01:33:15.933182+0000","last_clean":"2024-08-19T01:33:15.933182+0000","last_became_active":"2024-08-19T01:29:41.573885+0000","last_became_peered":"2024-08-19T01:29:41.573885+0000","last_unstale":"2024-08-19T01:33:15.933182+0000","last_undegraded":"2024-08-19T01:33:15.933182+0000","last_fullsized":"2024-08-19T01:33:15.933182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:26:37.926225+0000","last_clean_scrub_stamp":"2024-08-19T01:26:37.926225+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:55:38.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010021+0000","last_change":"2024-08-19T01:30:58.598415+0000","last_active":"2024-08-19T01:33:14.010021+0000","last_peered":"2024-08-19T01:33:14.010021+0000","last_clean":"2024-08-19T01:33:14.010021+0000","last_became_active":"2024-08-19T01:30:58.598086+0000","last_became_peered":"2024-08-19T01:30:58.598086+0000","last_unstale":"2024-08-19T01:33:14.010021+0000","last_undegraded":"2024-08-19T01:33:14.010021+0000","last_fullsized":"2024-08-19T01:33:14.010021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:21:37.338869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010133+0000","last_change":"2024-08-19T01:30:51.560407+0000","last_active":"2024-08-19T01:33:14.010133+0000","last_peered":"2024-08-19T01:33:14.010133+0000","last_clean":"2024-08-19T01:33:14.010133+0000","last_became_active":"2024-08-19T01:30:51.559881+0000","last_became_peered":"2024-08-19T01:30:51.559881+0000","last_unstale":"2024-08-19T01:33:14.010133+0000","last_undegraded":"2024-08-19T01:33:14.010133+0000","last_fullsized":"2024-08-19T01:33:14.010133+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:24:37.928254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822063+0000","last_change":"2024-08-19T01:30:49.711171+0000","last_active":"2024-08-19T01:33:13.822063+0000","last_peered":"2024-08-19T01:33:13.822063+0000","last_clean":"2024-08-19T01:33:13.822063+0000","last_became_active":"2024-08-19T01:30:49.710707+0000","last_became_peered":"2024-08-19T01:30:49.710707+0000","last_unstale":"2024-08-19T01:33:13.822063+0000","last_undegraded":"2024-08-19T01:33:13.822063+0000","last_fullsized":"2024-08-19T01:33:13.822063+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:58:20.924502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009117+0000","last_change":"2024-08-19T01:30:58.590829+0000","last_active":"2024-08-19T01:33:14.009117+0000","last_peered":"2024-08-19T01:33:14.009117+0000","last_clean":"2024-08-19T01:33:14.009117+0000","last_became_active":"2024-08-19T01:30:58.590460+0000","last_became_peered":"2024-08-19T01:30:58.590460+0000","last_unstale":"2024-08-19T01:33:14.009117+0000","last_undegraded":"2024-08-19T01:33:14.009117+0000","last_fullsized":"2024-08-19T01:33:14.009117+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:48:04.118550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009951+0000","last_change":"2024-08-19T01:30:51.564990+0000","last_active":"2024-08-19T01:33:14.009951+0000","last_peered":"2024-08-19T01:33:14.009951+0000","last_clean":"2024-08-19T01:33:14.009951+0000","last_became_active":"2024-08-19T01:30:51.564653+0000","last_became_peered":"2024-08-19T01:30:51.564653+0000","last_unstale":"2024-08-19T01:33:14.009951+0000","last_undegraded":"2024-08-19T01:33:14.009951+0000","last_fullsized":"2024-08-19T01:33:14.009951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:15:41.154376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":137,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:17.093439+0000","last_change":"2024-08-19T01:30:49.537637+0000","last_active":"2024-08-19T01:33:17.093439+0000","last_peered":"2024-08-19T01:33:17.093439+0000","last_clean":"2024-08-19T01:33:17.093439+0000","last_became_active":"2024-08-19T01:30:49.537277+0000","last_became_peered":"2024-08-19T01:30:49.537277+0000","last_unstale":"2024-08-19T01:33:17.093439+0000","last_undegraded":"2024-08-19T01:33:17.093439+0000","last_fullsized":"2024-08-19T01:33:17.093439+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:13.547597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010579+0000","last_change":"2024-08-19T01:30:58.594675+0000","last_active":"2024-08-19T01:33:14.010579+0000","last_peered":"2024-08-19T01:33:14.010579+0000","last_clean":"2024-08-19T01:33:14.010579+0000","last_became_active":"2024-08-19T01:30:58.594323+0000","last_became_peered":"2024-08-19T01:30:58.594323+0000","last_unstale":"2024-08-19T01:33:14.010579+0000","last_undegraded":"2024-08-19T01:33:14.010579+0000","last_fullsized":"2024-08-19T01:33:14.010579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:21:05.447106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010126+0000","last_change":"2024-08-19T01:30:51.565014+0000","last_active":"2024-08-19T01:33:14.010126+0000","last_peered":"2024-08-19T01:33:14.010126+0000","last_clean":"2024-08-19T01:33:14.010126+0000","last_became_active":"2024-08-19T01:30:51.564672+0000","last_became_peered":"2024-08-19T01:30:51.564672+0000","last_unstale":"2024-08-19T01:33:14.010126+0000","last_undegraded":"2024-08-19T01:33:14.010126+0000","last_fullsized":"2024-08-19T01:33:14.010126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:56:07.404042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010841+0000","last_change":"2024-08-19T01:30:49.538423+0000","last_active":"2024-08-19T01:33:14.010841+0000","last_peered":"2024-08-19T01:33:14.010841+0000","last_clean":"2024-08-19T01:33:14.010841+0000","last_became_active":"2024-08-19T01:30:49.538049+0000","last_became_peered":"2024-08-19T01:30:49.538049+0000","last_unstale":"2024-08-19T01:33:14.010841+0000","last_undegraded":"2024-08-19T01:33:14.010841+0000","last_fullsized":"2024-08-19T01:33:14.010841+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:27:41.449742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929000+0000","last_change":"2024-08-19T01:30:58.590963+0000","last_active":"2024-08-19T01:33:13.929000+0000","last_peered":"2024-08-19T01:33:13.929000+0000","last_clean":"2024-08-19T01:33:13.929000+0000","last_became_active":"2024-08-19T01:30:58.590584+0000","last_became_peered":"2024-08-19T01:30:58.590584+0000","last_unstale":"2024-08-19T01:33:13.929000+0000","last_undegraded":"2024-08-19T01:33:13.929000+0000","last_fullsized":"2024-08-19T01:33:13.929000+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:37.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009458+0000","last_change":"2024-08-19T01:30:51.561298+0000","last_active":"2024-08-19T01:33:14.009458+0000","last_peered":"2024-08-19T01:33:14.009458+0000","last_clean":"2024-08-19T01:33:14.009458+0000","last_became_active":"2024-08-19T01:30:51.560919+0000","last_became_peered":"2024-08-19T01:30:51.560919+0000","last_unstale":"2024-08-19T01:33:14.009458+0000","last_undegraded":"2024-08-19T01:33:14.009458+0000","last_fullsized":"2024-08-19T01:33:14.009458+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:04:39.808838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"94'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.761437+0000","last_change":"2024-08-19T01:30:49.541968+0000","last_active":"2024-08-19T01:33:14.761437+0000","last_peered":"2024-08-19T01:33:14.761437+0000","last_clean":"2024-08-19T01:33:14.761437+0000","last_became_active":"2024-08-19T01:30:49.541512+0000","last_became_peered":"2024-08-19T01:30:49.541512+0000","last_unstale":"2024-08-19T01:33:14.761437+0000","last_undegraded":"2024-08-19T01:33:14.761437+0000","last_fullsized":"2024-08-19T01:33:14.761437+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:34:28.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929302+0000","last_change":"2024-08-19T01:30:58.594776+0000","last_active":"2024-08-19T01:33:13.929302+0000","last_peered":"2024-08-19T01:33:13.929302+0000","last_clean":"2024-08-19T01:33:13.929302+0000","last_became_active":"2024-08-19T01:30:58.594414+0000","last_became_peered":"2024-08-19T01:30:58.594414+0000","last_unstale":"2024-08-19T01:33:13.929302+0000","last_undegraded":"2024-08-19T01:33:13.929302+0000","last_fullsized":"2024-08-19T01:33:13.929302+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:32:20.121504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929211+0000","last_change":"2024-08-19T01:30:51.556696+0000","last_active":"2024-08-19T01:33:13.929211+0000","last_peered":"2024-08-19T01:33:13.929211+0000","last_clean":"2024-08-19T01:33:13.929211+0000","last_became_active":"2024-08-19T01:30:51.555873+0000","last_became_peered":"2024-08-19T01:30:51.555873+0000","last_unstale":"2024-08-19T01:33:13.929211+0000","last_undegraded":"2024-08-19T01:33:13.929211+0000","last_fullsized":"2024-08-19T01:33:13.929211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:50:27.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920308+0000","last_change":"2024-08-19T01:30:49.576925+0000","last_active":"2024-08-19T01:33:13.920308+0000","last_peered":"2024-08-19T01:33:13.920308+0000","last_clean":"2024-08-19T01:33:13.920308+0000","last_became_active":"2024-08-19T01:30:49.576509+0000","last_became_peered":"2024-08-19T01:30:49.576509+0000","last_unstale":"2024-08-19T01:33:13.920308+0000","last_undegraded":"2024-08-19T01:33:13.920308+0000","last_fullsized":"2024-08-19T01:33:13.920308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:43:01.887020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010743+0000","last_change":"2024-08-19T01:32:35.826488+0000","last_active":"2024-08-19T01:33:14.010743+0000","last_peered":"2024-08-19T01:33:14.010743+0000","last_clean":"2024-08-19T01:33:14.010743+0000","last_became_active":"2024-08-19T01:32:35.826036+0000","last_became_peered":"2024-08-19T01:32:35.826036+0000","last_unstale":"2024-08-19T01:33:14.010743+0000","last_undegraded":"2024-08-19T01:33:14.010743+0000","last_fullsized":"2024-08-19T01:33:14.010743+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:16:03.388117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009567+0000","last_change":"2024-08-19T01:30:51.559622+0000","last_active":"2024-08-19T01:33:14.009567+0000","last_peered":"2024-08-19T01:33:14.009567+0000","last_clean":"2024-08-19T01:33:14.009567+0000","last_became_active":"2024-08-19T01:30:51.559263+0000","last_became_peered":"2024-08-19T01:30:51.559263+0000","last_unstale":"2024-08-19T01:33:14.009567+0000","last_undegraded":"2024-08-19T01:33:14.009567+0000","last_fullsized":"2024-08-19T01:33:14.009567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:11:25.758915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:18.744439+0000","last_change":"2024-08-19T01:30:49.540102+0000","last_active":"2024-08-19T01:33:18.744439+0000","last_peered":"2024-08-19T01:33:18.744439+0000","last_clean":"2024-08-19T01:33:18.744439+0000","last_became_active":"2024-08-19T01:30:49.539740+0000","last_became_peered":"2024-08-19T01:30:49.539740+0000","last_unstale":"2024-08-19T01:33:18.744439+0000","last_undegraded":"2024-08-19T01:33:18.744439+0000","last_fullsized":"2024-08-19T01:33:18.744439+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:47:21.684925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2289,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916713+0000","last_change":"2024-08-19T01:30:58.595748+0000","last_active":"2024-08-19T01:33:13.916713+0000","last_peered":"2024-08-19T01:33:13.916713+0000","last_clean":"2024-08-19T01:33:13.916713+0000","last_became_active":"2024-08-19T01:30:58.595396+0000","last_became_peered":"2024-08-19T01:30:58.595396+0000","last_unstale":"2024-08-19T01:33:13.916713+0000","last_undegraded":"2024-08-19T01:33:13.916713+0000","last_fullsized":"2024-08-19T01:33:13.916713+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:58:29.113687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009400+0000","last_change":"2024-08-19T01:30:51.559159+0000","last_active":"2024-08-19T01:33:14.009400+0000","last_peered":"2024-08-19T01:33:14.009400+0000","last_clean":"2024-08-19T01:33:14.009400+0000","last_became_active":"2024-08-19T01:30:51.558226+0000","last_became_peered":"2024-08-19T01:30:51.558226+0000","last_unstale":"2024-08-19T01:33:14.009400+0000","last_undegraded":"2024-08-19T01:33:14.009400+0000","last_fullsized":"2024-08-19T01:33:14.009400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:12:54.447301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929394+0000","last_change":"2024-08-19T01:30:49.540806+0000","last_active":"2024-08-19T01:33:13.929394+0000","last_peered":"2024-08-19T01:33:13.929394+0000","last_clean":"2024-08-19T01:33:13.929394+0000","last_became_active":"2024-08-19T01:30:49.540439+0000","last_became_peered":"2024-08-19T01:30:49.540439+0000","last_unstale":"2024-08-19T01:33:13.929394+0000","last_undegraded":"2024-08-19T01:33:13.929394+0000","last_fullsized":"2024-08-19T01:33:13.929394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:40.088842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009235+0000","last_change":"2024-08-19T01:30:58.592661+0000","last_active":"2024-08-19T01:33:14.009235+0000","last_peered":"2024-08-19T01:33:14.009235+0000","last_clean":"2024-08-19T01:33:14.009235+0000","last_became_active":"2024-08-19T01:30:58.592066+0000","last_became_peered":"2024-08-19T01:30:58.592066+0000","last_unstale":"2024-08-19T01:33:14.009235+0000","last_undegraded":"2024-08-19T01:33:14.009235+0000","last_fullsized":"2024-08-19T01:33:14.009235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:59:35.039682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009768+0000","last_change":"2024-08-19T01:30:51.558497+0000","last_active":"2024-08-19T01:33:14.009768+0000","last_peered":"2024-08-19T01:33:14.009768+0000","last_clean":"2024-08-19T01:33:14.009768+0000","last_became_active":"2024-08-19T01:30:51.558185+0000","last_became_peered":"2024-08-19T01:30:51.558185+0000","last_unstale":"2024-08-19T01:33:14.009768+0000","last_undegraded":"2024-08-19T01:33:14.009768+0000","last_fullsized":"2024-08-19T01:33:14.009768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:30:42.766269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821845+0000","last_change":"2024-08-19T01:30:49.539112+0000","last_active":"2024-08-19T01:33:13.821845+0000","last_peered":"2024-08-19T01:33:13.821845+0000","last_clean":"2024-08-19T01:33:13.821845+0000","last_became_active":"2024-08-19T01:30:49.538798+0000","last_became_peered":"2024-08-19T01:30:49.538798+0000","last_unstale":"2024-08-19T01:33:13.821845+0000","last_undegraded":"2024-08-19T01:33:13.821845+0000","last_fullsized":"2024-08-19T01:33:13.821845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:10:58.977041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929462+0000","last_change":"2024-08-19T01:30:58.590727+0000","last_active":"2024-08-19T01:33:13.929462+0000","last_peered":"2024-08-19T01:33:13.929462+0000","last_clean":"2024-08-19T01:33:13.929462+0000","last_became_active":"2024-08-19T01:30:58.590208+0000","last_became_peered":"2024-08-19T01:30:58.590208+0000","last_unstale":"2024-08-19T01:33:13.929462+0000","last_undegraded":"2024-08-19T01:33:13.929462+0000","last_fullsized":"2024-08-19T01:33:13.929462+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:22:36.422915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920646+0000","last_change":"2024-08-19T01:30:51.565113+0000","last_active":"2024-08-19T01:33:13.920646+0000","last_peered":"2024-08-19T01:33:13.920646+0000","last_clean":"2024-08-19T01:33:13.920646+0000","last_became_active":"2024-08-19T01:30:51.564769+0000","last_became_peered":"2024-08-19T01:30:51.564769+0000","last_unstale":"2024-08-19T01:33:13.920646+0000","last_undegraded":"2024-08-19T01:33:13.920646+0000","last_fullsized":"2024-08-19T01:33:13.920646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:13:28.373114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":325,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:19.745048+0000","last_change":"2024-08-19T01:31:35.610319+0000","last_active":"2024-08-19T01:33:19.745048+0000","last_peered":"2024-08-19T01:33:19.745048+0000","last_clean":"2024-08-19T01:33:19.745048+0000","last_became_active":"2024-08-19T01:31:35.602527+0000","last_became_peered":"2024-08-19T01:31:35.602527+0000","last_unstale":"2024-08-19T01:33:19.745048+0000","last_undegraded":"2024-08-19T01:33:19.745048+0000","last_fullsized":"2024-08-19T01:33:19.745048+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T01:50:05.511332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71690,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":222,"num_read_kb":179,"num_write":41,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3841,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.915395+0000","last_change":"2024-08-19T01:30:58.594836+0000","last_active":"2024-08-19T01:33:13.915395+0000","last_peered":"2024-08-19T01:33:13.915395+0000","last_clean":"2024-08-19T01:33:13.915395+0000","last_became_active":"2024-08-19T01:30:58.594441+0000","last_became_peered":"2024-08-19T01:30:58.594441+0000","last_unstale":"2024-08-19T01:33:13.915395+0000","last_undegraded":"2024-08-19T01:33:13.915395+0000","last_fullsized":"2024-08-19T01:33:13.915395+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:29:16.482181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009310+0000","last_change":"2024-08-19T01:30:51.562567+0000","last_active":"2024-08-19T01:33:14.009310+0000","last_peered":"2024-08-19T01:33:14.009310+0000","last_clean":"2024-08-19T01:33:14.009310+0000","last_became_active":"2024-08-19T01:30:51.562211+0000","last_became_peered":"2024-08-19T01:30:51.562211+0000","last_unstale":"2024-08-19T01:33:14.009310+0000","last_undegraded":"2024-08-19T01:33:14.009310+0000","last_fullsized":"2024-08-19T01:33:14.009310+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:14:58.193969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929607+0000","last_change":"2024-08-19T01:30:49.575957+0000","last_active":"2024-08-19T01:33:13.929607+0000","last_peered":"2024-08-19T01:33:13.929607+0000","last_clean":"2024-08-19T01:33:13.929607+0000","last_became_active":"2024-08-19T01:30:49.575570+0000","last_became_peered":"2024-08-19T01:30:49.575570+0000","last_unstale":"2024-08-19T01:33:13.929607+0000","last_undegraded":"2024-08-19T01:33:13.929607+0000","last_fullsized":"2024-08-19T01:33:13.929607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:47.038113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821913+0000","last_change":"2024-08-19T01:32:35.826212+0000","last_active":"2024-08-19T01:33:13.821913+0000","last_peered":"2024-08-19T01:33:13.821913+0000","last_clean":"2024-08-19T01:33:13.821913+0000","last_became_active":"2024-08-19T01:32:35.825748+0000","last_became_peered":"2024-08-19T01:32:35.825748+0000","last_unstale":"2024-08-19T01:33:13.821913+0000","last_undegraded":"2024-08-19T01:33:13.821913+0000","last_fullsized":"2024-08-19T01:33:13.821913+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:34:59.072131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011992+0000","last_change":"2024-08-19T01:31:35.606449+0000","last_active":"2024-08-19T01:33:14.011992+0000","last_peered":"2024-08-19T01:33:14.011992+0000","last_clean":"2024-08-19T01:33:14.011992+0000","last_became_active":"2024-08-19T01:31:35.606105+0000","last_became_peered":"2024-08-19T01:31:35.606105+0000","last_unstale":"2024-08-19T01:33:14.011992+0000","last_undegraded":"2024-08-19T01:33:14.011992+0000","last_fullsized":"2024-08-19T01:33:14.011992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:36:21.834763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010069+0000","last_change":"2024-08-19T01:30:49.542441+0000","last_active":"2024-08-19T01:33:14.010069+0000","last_peered":"2024-08-19T01:33:14.010069+0000","last_clean":"2024-08-19T01:33:14.010069+0000","last_became_active":"2024-08-19T01:30:49.542106+0000","last_became_peered":"2024-08-19T01:30:49.542106+0000","last_unstale":"2024-08-19T01:33:14.010069+0000","last_undegraded":"2024-08-19T01:33:14.010069+0000","last_fullsized":"2024-08-19T01:33:14.010069+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:21:01.956282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914543+0000","last_change":"2024-08-19T01:30:58.588899+0000","last_active":"2024-08-19T01:33:13.914543+0000","last_peered":"2024-08-19T01:33:13.914543+0000","last_clean":"2024-08-19T01:33:13.914543+0000","last_became_active":"2024-08-19T01:30:58.588521+0000","last_became_peered":"2024-08-19T01:30:58.588521+0000","last_unstale":"2024-08-19T01:33:13.914543+0000","last_undegraded":"2024-08-19T01:33:13.914543+0000","last_fullsized":"2024-08-19T01:33:13.914543+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:43:59.485456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009811+0000","last_change":"2024-08-19T01:30:51.557248+0000","last_active":"2024-08-19T01:33:14.009811+0000","last_peered":"2024-08-19T01:33:14.009811+0000","last_clean":"2024-08-19T01:33:14.009811+0000","last_became_active":"2024-08-19T01:30:51.556448+0000","last_became_peered":"2024-08-19T01:30:51.556448+0000","last_unstale":"2024-08-19T01:33:14.009811+0000","last_undegraded":"2024-08-19T01:33:14.009811+0000","last_fullsized":"2024-08-19T01:33:14.009811+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:19:19.587955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:19.745202+0000","last_change":"2024-08-19T01:30:49.540360+0000","last_active":"2024-08-19T01:33:19.745202+0000","last_peered":"2024-08-19T01:33:19.745202+0000","last_clean":"2024-08-19T01:33:19.745202+0000","last_became_active":"2024-08-19T01:30:49.539993+0000","last_became_peered":"2024-08-19T01:30:49.539993+0000","last_unstale":"2024-08-19T01:33:19.745202+0000","last_undegraded":"2024-08-19T01:33:19.745202+0000","last_fullsized":"2024-08-19T01:33:19.745202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:05:28.308441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820525+0000","last_change":"2024-08-19T01:30:58.592387+0000","last_active":"2024-08-19T01:33:13.820525+0000","last_peered":"2024-08-19T01:33:13.820525+0000","last_clean":"2024-08-19T01:33:13.820525+0000","last_became_active":"2024-08-19T01:30:58.592031+0000","last_became_peered":"2024-08-19T01:30:58.592031+0000","last_unstale":"2024-08-19T01:33:13.820525+0000","last_undegraded":"2024-08-19T01:33:13.820525+0000","last_fullsized":"2024-08-19T01:33:13.820525+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:31:23.159012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011277+0000","last_change":"2024-08-19T01:30:58.595650+0000","last_active":"2024-08-19T01:33:14.011277+0000","last_peered":"2024-08-19T01:33:14.011277+0000","last_clean":"2024-08-19T01:33:14.011277+0000","last_became_active":"2024-08-19T01:30:58.595304+0000","last_became_peered":"2024-08-19T01:30:58.595304+0000","last_unstale":"2024-08-19T01:33:14.011277+0000","last_undegraded":"2024-08-19T01:33:14.011277+0000","last_fullsized":"2024-08-19T01:33:14.011277+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:23:53.517355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.914983+0000","last_change":"2024-08-19T01:30:51.548683+0000","last_active":"2024-08-19T01:33:13.914983+0000","last_peered":"2024-08-19T01:33:13.914983+0000","last_clean":"2024-08-19T01:33:13.914983+0000","last_became_active":"2024-08-19T01:30:51.548382+0000","last_became_peered":"2024-08-19T01:30:51.548382+0000","last_unstale":"2024-08-19T01:33:13.914983+0000","last_undegraded":"2024-08-19T01:33:13.914983+0000","last_fullsized":"2024-08-19T01:33:13.914983+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:46:37.551017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929237+0000","last_change":"2024-08-19T01:30:49.540136+0000","last_active":"2024-08-19T01:33:13.929237+0000","last_peered":"2024-08-19T01:33:13.929237+0000","last_clean":"2024-08-19T01:33:13.929237+0000","last_became_active":"2024-08-19T01:30:49.539680+0000","last_became_peered":"2024-08-19T01:30:49.539680+0000","last_unstale":"2024-08-19T01:33:13.929237+0000","last_undegraded":"2024-08-19T01:33:13.929237+0000","last_fullsized":"2024-08-19T01:33:13.929237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:00:25.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009451+0000","last_change":"2024-08-19T01:30:58.597130+0000","last_active":"2024-08-19T01:33:14.009451+0000","last_peered":"2024-08-19T01:33:14.009451+0000","last_clean":"2024-08-19T01:33:14.009451+0000","last_became_active":"2024-08-19T01:30:58.596834+0000","last_became_peered":"2024-08-19T01:30:58.596834+0000","last_unstale":"2024-08-19T01:33:14.009451+0000","last_undegraded":"2024-08-19T01:33:14.009451+0000","last_fullsized":"2024-08-19T01:33:14.009451+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:13:10.441421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820503+0000","last_change":"2024-08-19T01:30:51.555575+0000","last_active":"2024-08-19T01:33:13.820503+0000","last_peered":"2024-08-19T01:33:13.820503+0000","last_clean":"2024-08-19T01:33:13.820503+0000","last_became_active":"2024-08-19T01:30:51.555309+0000","last_became_peered":"2024-08-19T01:30:51.555309+0000","last_unstale":"2024-08-19T01:33:13.820503+0000","last_undegraded":"2024-08-19T01:33:13.820503+0000","last_fullsized":"2024-08-19T01:33:13.820503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:11:25.515286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009505+0000","last_change":"2024-08-19T01:30:49.536844+0000","last_active":"2024-08-19T01:33:14.009505+0000","last_peered":"2024-08-19T01:33:14.009505+0000","last_clean":"2024-08-19T01:33:14.009505+0000","last_became_active":"2024-08-19T01:30:49.536496+0000","last_became_peered":"2024-08-19T01:30:49.536496+0000","last_unstale":"2024-08-19T01:33:14.009505+0000","last_undegraded":"2024-08-19T01:33:14.009505+0000","last_fullsized":"2024-08-19T01:33:14.009505+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:06:04.917188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932470+0000","last_change":"2024-08-19T01:30:49.539356+0000","last_active":"2024-08-19T01:33:13.932470+0000","last_peered":"2024-08-19T01:33:13.932470+0000","last_clean":"2024-08-19T01:33:13.932470+0000","last_became_active":"2024-08-19T01:30:49.538953+0000","last_became_peered":"2024-08-19T01:30:49.538953+0000","last_unstale":"2024-08-19T01:33:13.932470+0000","last_undegraded":"2024-08-19T01:33:13.932470+0000","last_fullsized":"2024-08-19T01:33:13.932470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:13:17.845140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929202+0000","last_change":"2024-08-19T01:31:35.870021+0000","last_active":"2024-08-19T01:33:13.929202+0000","last_peered":"2024-08-19T01:33:13.929202+0000","last_clean":"2024-08-19T01:33:13.929202+0000","last_became_active":"2024-08-19T01:31:35.869742+0000","last_became_peered":"2024-08-19T01:31:35.869742+0000","last_unstale":"2024-08-19T01:33:13.929202+0000","last_undegraded":"2024-08-19T01:33:13.929202+0000","last_fullsized":"2024-08-19T01:33:13.929202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:31:51.848694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821878+0000","last_change":"2024-08-19T01:30:58.590385+0000","last_active":"2024-08-19T01:33:13.821878+0000","last_peered":"2024-08-19T01:33:13.821878+0000","last_clean":"2024-08-19T01:33:13.821878+0000","last_became_active":"2024-08-19T01:30:58.590004+0000","last_became_peered":"2024-08-19T01:30:58.590004+0000","last_unstale":"2024-08-19T01:33:13.821878+0000","last_undegraded":"2024-08-19T01:33:13.821878+0000","last_fullsized":"2024-08-19T01:33:13.821878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:13:01.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":172,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:18.744558+0000","last_change":"2024-08-19T01:31:35.614962+0000","last_active":"2024-08-19T01:33:18.744558+0000","last_peered":"2024-08-19T01:33:18.744558+0000","last_clean":"2024-08-19T01:33:18.744558+0000","last_became_active":"2024-08-19T01:31:35.605904+0000","last_became_peered":"2024-08-19T01:31:35.605904+0000","last_unstale":"2024-08-19T01:33:18.744558+0000","last_undegraded":"2024-08-19T01:33:18.744558+0000","last_fullsized":"2024-08-19T01:33:18.744558+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:48.510365+0000","last_clean_scrub_stamp":"2024-08-19T01:30:48.510365+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:26:36.102805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920402+0000","last_change":"2024-08-19T01:30:51.550448+0000","last_active":"2024-08-19T01:33:13.920402+0000","last_peered":"2024-08-19T01:33:13.920402+0000","last_clean":"2024-08-19T01:33:13.920402+0000","last_became_active":"2024-08-19T01:30:51.550074+0000","last_became_peered":"2024-08-19T01:30:51.550074+0000","last_unstale":"2024-08-19T01:33:13.920402+0000","last_undegraded":"2024-08-19T01:33:13.920402+0000","last_fullsized":"2024-08-19T01:33:13.920402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:33:13.016183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822063+0000","last_change":"2024-08-19T01:30:58.592077+0000","last_active":"2024-08-19T01:33:13.822063+0000","last_peered":"2024-08-19T01:33:13.822063+0000","last_clean":"2024-08-19T01:33:13.822063+0000","last_became_active":"2024-08-19T01:30:58.591651+0000","last_became_peered":"2024-08-19T01:30:58.591651+0000","last_unstale":"2024-08-19T01:33:13.822063+0000","last_undegraded":"2024-08-19T01:33:13.822063+0000","last_fullsized":"2024-08-19T01:33:13.822063+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:57.560443+0000","last_clean_scrub_stamp":"2024-08-19T01:30:57.560443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:09:24.980954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009090+0000","last_change":"2024-08-19T01:30:51.552772+0000","last_active":"2024-08-19T01:33:14.009090+0000","last_peered":"2024-08-19T01:33:14.009090+0000","last_clean":"2024-08-19T01:33:14.009090+0000","last_became_active":"2024-08-19T01:30:51.550904+0000","last_became_peered":"2024-08-19T01:30:51.550904+0000","last_unstale":"2024-08-19T01:33:14.009090+0000","last_undegraded":"2024-08-19T01:33:14.009090+0000","last_fullsized":"2024-08-19T01:33:14.009090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:21:50.082587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821870+0000","last_change":"2024-08-19T01:30:51.559756+0000","last_active":"2024-08-19T01:33:13.821870+0000","last_peered":"2024-08-19T01:33:13.821870+0000","last_clean":"2024-08-19T01:33:13.821870+0000","last_became_active":"2024-08-19T01:30:51.559370+0000","last_became_peered":"2024-08-19T01:30:51.559370+0000","last_unstale":"2024-08-19T01:33:13.821870+0000","last_undegraded":"2024-08-19T01:33:13.821870+0000","last_fullsized":"2024-08-19T01:33:13.821870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:27:30.525346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821589+0000","last_change":"2024-08-19T01:30:51.556577+0000","last_active":"2024-08-19T01:33:13.821589+0000","last_peered":"2024-08-19T01:33:13.821589+0000","last_clean":"2024-08-19T01:33:13.821589+0000","last_became_active":"2024-08-19T01:30:51.555929+0000","last_became_peered":"2024-08-19T01:30:51.555929+0000","last_unstale":"2024-08-19T01:33:13.821589+0000","last_undegraded":"2024-08-19T01:33:13.821589+0000","last_fullsized":"2024-08-19T01:33:13.821589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T04:46:31.660752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010627+0000","last_change":"2024-08-19T01:30:51.558626+0000","last_active":"2024-08-19T01:33:14.010627+0000","last_peered":"2024-08-19T01:33:14.010627+0000","last_clean":"2024-08-19T01:33:14.010627+0000","last_became_active":"2024-08-19T01:30:51.558188+0000","last_became_peered":"2024-08-19T01:30:51.558188+0000","last_unstale":"2024-08-19T01:33:14.010627+0000","last_undegraded":"2024-08-19T01:33:14.010627+0000","last_fullsized":"2024-08-19T01:33:14.010627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:43:44.777643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010095+0000","last_change":"2024-08-19T01:30:51.552870+0000","last_active":"2024-08-19T01:33:14.010095+0000","last_peered":"2024-08-19T01:33:14.010095+0000","last_clean":"2024-08-19T01:33:14.010095+0000","last_became_active":"2024-08-19T01:30:51.552514+0000","last_became_peered":"2024-08-19T01:30:51.552514+0000","last_unstale":"2024-08-19T01:33:14.010095+0000","last_undegraded":"2024-08-19T01:33:14.010095+0000","last_fullsized":"2024-08-19T01:33:14.010095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:10:58.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009491+0000","last_change":"2024-08-19T01:30:51.554216+0000","last_active":"2024-08-19T01:33:14.009491+0000","last_peered":"2024-08-19T01:33:14.009491+0000","last_clean":"2024-08-19T01:33:14.009491+0000","last_became_active":"2024-08-19T01:30:51.552535+0000","last_became_peered":"2024-08-19T01:30:51.552535+0000","last_unstale":"2024-08-19T01:33:14.009491+0000","last_undegraded":"2024-08-19T01:33:14.009491+0000","last_fullsized":"2024-08-19T01:33:14.009491+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:26:52.971897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822254+0000","last_change":"2024-08-19T01:30:51.561703+0000","last_active":"2024-08-19T01:33:13.822254+0000","last_peered":"2024-08-19T01:33:13.822254+0000","last_clean":"2024-08-19T01:33:13.822254+0000","last_became_active":"2024-08-19T01:30:51.561385+0000","last_became_peered":"2024-08-19T01:30:51.561385+0000","last_unstale":"2024-08-19T01:33:13.822254+0000","last_undegraded":"2024-08-19T01:33:13.822254+0000","last_fullsized":"2024-08-19T01:33:13.822254+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:07:23.187023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820799+0000","last_change":"2024-08-19T01:30:51.564047+0000","last_active":"2024-08-19T01:33:13.820799+0000","last_peered":"2024-08-19T01:33:13.820799+0000","last_clean":"2024-08-19T01:33:13.820799+0000","last_became_active":"2024-08-19T01:30:51.563743+0000","last_became_peered":"2024-08-19T01:30:51.563743+0000","last_unstale":"2024-08-19T01:33:13.820799+0000","last_undegraded":"2024-08-19T01:33:13.820799+0000","last_fullsized":"2024-08-19T01:33:13.820799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:50:37.639496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.929073+0000","last_change":"2024-08-19T01:30:51.557058+0000","last_active":"2024-08-19T01:33:13.929073+0000","last_peered":"2024-08-19T01:33:13.929073+0000","last_clean":"2024-08-19T01:33:13.929073+0000","last_became_active":"2024-08-19T01:30:51.554611+0000","last_became_peered":"2024-08-19T01:30:51.554611+0000","last_unstale":"2024-08-19T01:33:13.929073+0000","last_undegraded":"2024-08-19T01:33:13.929073+0000","last_fullsized":"2024-08-19T01:33:13.929073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:56:25.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.820511+0000","last_change":"2024-08-19T01:30:51.564119+0000","last_active":"2024-08-19T01:33:13.820511+0000","last_peered":"2024-08-19T01:33:13.820511+0000","last_clean":"2024-08-19T01:33:13.820511+0000","last_became_active":"2024-08-19T01:30:51.563840+0000","last_became_peered":"2024-08-19T01:30:51.563840+0000","last_unstale":"2024-08-19T01:33:13.820511+0000","last_undegraded":"2024-08-19T01:33:13.820511+0000","last_fullsized":"2024-08-19T01:33:13.820511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T11:44:44.928760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009253+0000","last_change":"2024-08-19T01:30:51.561639+0000","last_active":"2024-08-19T01:33:14.009253+0000","last_peered":"2024-08-19T01:33:14.009253+0000","last_clean":"2024-08-19T01:33:14.009253+0000","last_became_active":"2024-08-19T01:30:51.561207+0000","last_became_peered":"2024-08-19T01:30:51.561207+0000","last_unstale":"2024-08-19T01:33:14.009253+0000","last_undegraded":"2024-08-19T01:33:14.009253+0000","last_fullsized":"2024-08-19T01:33:14.009253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:12:20.076628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011400+0000","last_change":"2024-08-19T01:30:51.556700+0000","last_active":"2024-08-19T01:33:14.011400+0000","last_peered":"2024-08-19T01:33:14.011400+0000","last_clean":"2024-08-19T01:33:14.011400+0000","last_became_active":"2024-08-19T01:30:51.556407+0000","last_became_peered":"2024-08-19T01:30:51.556407+0000","last_unstale":"2024-08-19T01:33:14.011400+0000","last_undegraded":"2024-08-19T01:33:14.011400+0000","last_fullsized":"2024-08-19T01:33:14.011400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:03:16.664253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009274+0000","last_change":"2024-08-19T01:30:51.553764+0000","last_active":"2024-08-19T01:33:14.009274+0000","last_peered":"2024-08-19T01:33:14.009274+0000","last_clean":"2024-08-19T01:33:14.009274+0000","last_became_active":"2024-08-19T01:30:51.553442+0000","last_became_peered":"2024-08-19T01:30:51.553442+0000","last_unstale":"2024-08-19T01:33:14.009274+0000","last_undegraded":"2024-08-19T01:33:14.009274+0000","last_fullsized":"2024-08-19T01:33:14.009274+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:36:44.476979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.822255+0000","last_change":"2024-08-19T01:30:51.565528+0000","last_active":"2024-08-19T01:33:13.822255+0000","last_peered":"2024-08-19T01:33:13.822255+0000","last_clean":"2024-08-19T01:33:13.822255+0000","last_became_active":"2024-08-19T01:30:51.564649+0000","last_became_peered":"2024-08-19T01:30:51.564649+0000","last_unstale":"2024-08-19T01:33:13.822255+0000","last_undegraded":"2024-08-19T01:33:13.822255+0000","last_fullsized":"2024-08-19T01:33:13.822255+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:04:12.868496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.917383+0000","last_change":"2024-08-19T01:30:51.553425+0000","last_active":"2024-08-19T01:33:13.917383+0000","last_peered":"2024-08-19T01:33:13.917383+0000","last_clean":"2024-08-19T01:33:13.917383+0000","last_became_active":"2024-08-19T01:30:51.553123+0000","last_became_peered":"2024-08-19T01:30:51.553123+0000","last_unstale":"2024-08-19T01:33:13.917383+0000","last_undegraded":"2024-08-19T01:33:13.917383+0000","last_fullsized":"2024-08-19T01:33:13.917383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:09:07.969228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009636+0000","last_change":"2024-08-19T01:30:51.556820+0000","last_active":"2024-08-19T01:33:14.009636+0000","last_peered":"2024-08-19T01:33:14.009636+0000","last_clean":"2024-08-19T01:33:14.009636+0000","last_became_active":"2024-08-19T01:30:51.556532+0000","last_became_peered":"2024-08-19T01:30:51.556532+0000","last_unstale":"2024-08-19T01:33:14.009636+0000","last_undegraded":"2024-08-19T01:33:14.009636+0000","last_fullsized":"2024-08-19T01:33:14.009636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:40:37.053193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.008986+0000","last_change":"2024-08-19T01:30:51.561610+0000","last_active":"2024-08-19T01:33:14.008986+0000","last_peered":"2024-08-19T01:33:14.008986+0000","last_clean":"2024-08-19T01:33:14.008986+0000","last_became_active":"2024-08-19T01:30:51.561270+0000","last_became_peered":"2024-08-19T01:30:51.561270+0000","last_unstale":"2024-08-19T01:33:14.008986+0000","last_undegraded":"2024-08-19T01:33:14.008986+0000","last_fullsized":"2024-08-19T01:33:14.008986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T09:15:15.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010003+0000","last_change":"2024-08-19T01:30:51.558128+0000","last_active":"2024-08-19T01:33:14.010003+0000","last_peered":"2024-08-19T01:33:14.010003+0000","last_clean":"2024-08-19T01:33:14.010003+0000","last_became_active":"2024-08-19T01:30:51.557785+0000","last_became_peered":"2024-08-19T01:30:51.557785+0000","last_unstale":"2024-08-19T01:33:14.010003+0000","last_undegraded":"2024-08-19T01:33:14.010003+0000","last_fullsized":"2024-08-19T01:33:14.010003+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:22:14.849212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010269+0000","last_change":"2024-08-19T01:30:51.553917+0000","last_active":"2024-08-19T01:33:14.010269+0000","last_peered":"2024-08-19T01:33:14.010269+0000","last_clean":"2024-08-19T01:33:14.010269+0000","last_became_active":"2024-08-19T01:30:51.553596+0000","last_became_peered":"2024-08-19T01:30:51.553596+0000","last_unstale":"2024-08-19T01:33:14.010269+0000","last_undegraded":"2024-08-19T01:33:14.010269+0000","last_fullsized":"2024-08-19T01:33:14.010269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:03:36.001486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011300+0000","last_change":"2024-08-19T01:30:51.555201+0000","last_active":"2024-08-19T01:33:14.011300+0000","last_peered":"2024-08-19T01:33:14.011300+0000","last_clean":"2024-08-19T01:33:14.011300+0000","last_became_active":"2024-08-19T01:30:51.554160+0000","last_became_peered":"2024-08-19T01:30:51.554160+0000","last_unstale":"2024-08-19T01:33:14.011300+0000","last_undegraded":"2024-08-19T01:33:14.011300+0000","last_fullsized":"2024-08-19T01:33:14.011300+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:58:20.586931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821883+0000","last_change":"2024-08-19T01:30:51.560517+0000","last_active":"2024-08-19T01:33:13.821883+0000","last_peered":"2024-08-19T01:33:13.821883+0000","last_clean":"2024-08-19T01:33:13.821883+0000","last_became_active":"2024-08-19T01:30:51.557454+0000","last_became_peered":"2024-08-19T01:30:51.557454+0000","last_unstale":"2024-08-19T01:33:13.821883+0000","last_undegraded":"2024-08-19T01:33:13.821883+0000","last_fullsized":"2024-08-19T01:33:13.821883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:41:08.192165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.010185+0000","last_change":"2024-08-19T01:30:51.565961+0000","last_active":"2024-08-19T01:33:14.010185+0000","last_peered":"2024-08-19T01:33:14.010185+0000","last_clean":"2024-08-19T01:33:14.010185+0000","last_became_active":"2024-08-19T01:30:51.565491+0000","last_became_peered":"2024-08-19T01:30:51.565491+0000","last_unstale":"2024-08-19T01:33:14.010185+0000","last_undegraded":"2024-08-19T01:33:14.010185+0000","last_fullsized":"2024-08-19T01:33:14.010185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T03:53:06.263411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932802+0000","last_change":"2024-08-19T01:30:51.551297+0000","last_active":"2024-08-19T01:33:13.932802+0000","last_peered":"2024-08-19T01:33:13.932802+0000","last_clean":"2024-08-19T01:33:13.932802+0000","last_became_active":"2024-08-19T01:30:51.550929+0000","last_became_peered":"2024-08-19T01:30:51.550929+0000","last_unstale":"2024-08-19T01:33:13.932802+0000","last_undegraded":"2024-08-19T01:33:13.932802+0000","last_fullsized":"2024-08-19T01:33:13.932802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:07:03.625360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.916372+0000","last_change":"2024-08-19T01:31:35.602883+0000","last_active":"2024-08-19T01:33:13.916372+0000","last_peered":"2024-08-19T01:33:13.916372+0000","last_clean":"2024-08-19T01:33:13.916372+0000","last_became_active":"2024-08-19T01:31:35.602571+0000","last_became_peered":"2024-08-19T01:31:35.602571+0000","last_unstale":"2024-08-19T01:33:13.916372+0000","last_undegraded":"2024-08-19T01:33:13.916372+0000","last_fullsized":"2024-08-19T01:33:13.916372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:08:13.359603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821061+0000","last_change":"2024-08-19T01:30:51.552548+0000","last_active":"2024-08-19T01:33:13.821061+0000","last_peered":"2024-08-19T01:33:13.821061+0000","last_clean":"2024-08-19T01:33:13.821061+0000","last_became_active":"2024-08-19T01:30:51.552264+0000","last_became_peered":"2024-08-19T01:30:51.552264+0000","last_unstale":"2024-08-19T01:33:13.821061+0000","last_undegraded":"2024-08-19T01:33:13.821061+0000","last_fullsized":"2024-08-19T01:33:13.821061+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T06:17:53.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009360+0000","last_change":"2024-08-19T01:30:51.558686+0000","last_active":"2024-08-19T01:33:14.009360+0000","last_peered":"2024-08-19T01:33:14.009360+0000","last_clean":"2024-08-19T01:33:14.009360+0000","last_became_active":"2024-08-19T01:30:51.558190+0000","last_became_peered":"2024-08-19T01:30:51.558190+0000","last_unstale":"2024-08-19T01:33:14.009360+0000","last_undegraded":"2024-08-19T01:33:14.009360+0000","last_fullsized":"2024-08-19T01:33:14.009360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:52:33.056948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.932373+0000","last_change":"2024-08-19T01:30:51.548868+0000","last_active":"2024-08-19T01:33:13.932373+0000","last_peered":"2024-08-19T01:33:13.932373+0000","last_clean":"2024-08-19T01:33:13.932373+0000","last_became_active":"2024-08-19T01:30:51.548347+0000","last_became_peered":"2024-08-19T01:30:51.548347+0000","last_unstale":"2024-08-19T01:33:13.932373+0000","last_undegraded":"2024-08-19T01:33:13.932373+0000","last_fullsized":"2024-08-19T01:33:13.932373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T05:39:22.049290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.920367+0000","last_change":"2024-08-19T01:30:51.554054+0000","last_active":"2024-08-19T01:33:13.920367+0000","last_peered":"2024-08-19T01:33:13.920367+0000","last_clean":"2024-08-19T01:33:13.920367+0000","last_became_active":"2024-08-19T01:30:51.553731+0000","last_became_peered":"2024-08-19T01:30:51.553731+0000","last_unstale":"2024-08-19T01:33:13.920367+0000","last_undegraded":"2024-08-19T01:33:13.920367+0000","last_fullsized":"2024-08-19T01:33:13.920367+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T10:25:17.939024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:13.821646+0000","last_change":"2024-08-19T01:30:51.558662+0000","last_active":"2024-08-19T01:33:13.821646+0000","last_peered":"2024-08-19T01:33:13.821646+0000","last_clean":"2024-08-19T01:33:13.821646+0000","last_became_active":"2024-08-19T01:30:51.558131+0000","last_became_peered":"2024-08-19T01:30:51.558131+0000","last_unstale":"2024-08-19T01:33:13.821646+0000","last_undegraded":"2024-08-19T01:33:13.821646+0000","last_fullsized":"2024-08-19T01:33:13.821646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T02:31:25.054005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011016+0000","last_change":"2024-08-19T01:30:51.565170+0000","last_active":"2024-08-19T01:33:14.011016+0000","last_peered":"2024-08-19T01:33:14.011016+0000","last_clean":"2024-08-19T01:33:14.011016+0000","last_became_active":"2024-08-19T01:30:51.564765+0000","last_became_peered":"2024-08-19T01:30:51.564765+0000","last_unstale":"2024-08-19T01:33:14.011016+0000","last_undegraded":"2024-08-19T01:33:14.011016+0000","last_fullsized":"2024-08-19T01:33:14.011016+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:00:48.668865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.011370+0000","last_change":"2024-08-19T01:30:51.555300+0000","last_active":"2024-08-19T01:33:14.011370+0000","last_peered":"2024-08-19T01:33:14.011370+0000","last_clean":"2024-08-19T01:33:14.011370+0000","last_became_active":"2024-08-19T01:30:51.554977+0000","last_became_peered":"2024-08-19T01:30:51.554977+0000","last_unstale":"2024-08-19T01:33:14.011370+0000","last_undegraded":"2024-08-19T01:33:14.011370+0000","last_fullsized":"2024-08-19T01:33:14.011370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T08:18:58.961783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"93'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-19T01:33:14.009590+0000","last_change":"2024-08-19T01:30:51.557906+0000","last_active":"2024-08-19T01:33:14.009590+0000","last_peered":"2024-08-19T01:33:14.009590+0000","last_clean":"2024-08-19T01:33:14.009590+0000","last_became_active":"2024-08-19T01:30:51.557598+0000","last_became_peered":"2024-08-19T01:30:51.557598+0000","last_unstale":"2024-08-19T01:33:14.009590+0000","last_undegraded":"2024-08-19T01:33:14.009590+0000","last_fullsized":"2024-08-19T01:33:14.009590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T01:30:50.525779+0000","last_clean_scrub_stamp":"2024-08-19T01:30:50.525779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T07:33:12.785910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799760,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1013,"num_read_kb":722,"num_write":431,"num_write_kb":2366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4463,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":2399280,"data_compressed":103473,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":317,"ondisk_log_size":317,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107877,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481834,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":542338,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743471,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36560,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708592,"statfs":{"total":95995035648,"available":95957598208,"internally_reserved":0,"allocated":1245184,"data_stored":1535325,"data_compressed":83646,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906996,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":935790,"data_compressed":51774,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168631,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":935710,"data_compressed":51774,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430269,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":1141024,"data_compressed":68444,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593794,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":537750,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822727,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":1463143,"data_compressed":83646,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986251,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32280,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712872,"statfs":{"total":95995035648,"available":95961980928,"internally_reserved":0,"allocated":1130496,"data_stored":615791,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215182,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":1259392,"data_compressed":66976,"data_compressed_allocated":90112,"data_compressed_original":790528,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":538388,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":611283,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":539138,"data_compressed":32485,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35959,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":73623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":331019,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":403201,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T01:33:22.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:33:22.858 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:23.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:22 smithi064 ceph-mon[18515]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-08-19T01:33:23.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:22 smithi064 ceph-mon[18515]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:23.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:22 smithi123 ceph-mon[21829]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-08-19T01:33:23.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:22 smithi123 ceph-mon[21829]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:23.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:22 smithi161 ceph-mon[21757]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-08-19T01:33:23.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:22 smithi161 ceph-mon[21757]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 -- 172.21.15.64:0/205411199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5941018b0 msgr2=0x7fb594101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 --2- 172.21.15.64:0/205411199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5941018b0 0x7fb594101d10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb584009930 tx=0x7fb58402a420 comp rx=0 tx=0).stop 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 -- 172.21.15.64:0/205411199 shutdown_connections 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 --2- 172.21.15.64:0/205411199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb594102250 0x7fb59410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 --2- 172.21.15.64:0/205411199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5941018b0 0x7fb594101d10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 --2- 172.21.15.64:0/205411199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5941078b0 0x7fb594107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 -- 172.21.15.64:0/205411199 >> 172.21.15.64:0/205411199 conn(0x7fb5940fd5f0 msgr2=0x7fb5940ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 -- 172.21.15.64:0/205411199 shutdown_connections 2024-08-19T01:33:23.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.311+0000 7fb59c484640 1 -- 172.21.15.64:0/205411199 wait complete. 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 Processor -- start 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 -- start start 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5941018b0 0x7fb5941981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb594102250 0x7fb594198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5941078b0 0x7fb59419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb594111cb0 con 0x7fb594102250 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb594111b30 con 0x7fb5941078b0 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59c484640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb594111e30 con 0x7fb5941018b0 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb599a1f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb594102250 0x7fb594198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb599a1f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb594102250 0x7fb594198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50898/0 (socket says 172.21.15.64:50898) 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb599a1f640 1 -- 172.21.15.64:0/3632795973 learned_addr learned my addr 172.21.15.64:0/3632795973 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59a220640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5941018b0 0x7fb5941981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:23.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.312+0000 7fb59aa21640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5941078b0 0x7fb59419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb599a1f640 1 -- 172.21.15.64:0/3632795973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5941018b0 msgr2=0x7fb5941981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb599a1f640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5941018b0 0x7fb5941981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb599a1f640 1 -- 172.21.15.64:0/3632795973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5941078b0 msgr2=0x7fb59419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb599a1f640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5941078b0 0x7fb59419caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb599a1f640 1 -- 172.21.15.64:0/3632795973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59419d180 con 0x7fb594102250 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb59aa21640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5941078b0 0x7fb59419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb599a1f640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb594102250 0x7fb594198710 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb584009230 tx=0x7fb584004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb58b7fe640 1 -- 172.21.15.64:0/3632795973 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58403f030 con 0x7fb594102250 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59419d410 con 0x7fb594102250 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb58b7fe640 1 -- 172.21.15.64:0/3632795973 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb584003ad0 con 0x7fb594102250 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb58b7fe640 1 -- 172.21.15.64:0/3632795973 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584035070 con 0x7fb594102250 2024-08-19T01:33:23.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.313+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5941a4d50 con 0x7fb594102250 2024-08-19T01:33:23.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.314+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb594101d10 con 0x7fb594102250 2024-08-19T01:33:23.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.316+0000 7fb58b7fe640 1 -- 172.21.15.64:0/3632795973 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fb584042030 con 0x7fb594102250 2024-08-19T01:33:23.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.316+0000 7fb58b7fe640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb5800a2e10 0x7fb5800a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:23.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.317+0000 7fb59a220640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb5800a2e10 0x7fb5800a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:23.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.317+0000 7fb58b7fe640 1 -- 172.21.15.64:0/3632795973 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb5840f0370 con 0x7fb594102250 2024-08-19T01:33:23.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.317+0000 7fb59a220640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb5800a2e10 0x7fb5800a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb59000bb60 tx=0x7fb59000b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:23.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.320+0000 7fb58b7fe640 1 -- 172.21.15.64:0/3632795973 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb5840bca00 con 0x7fb594102250 2024-08-19T01:33:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.530+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb594199530 con 0x7fb594102250 2024-08-19T01:33:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.530+0000 7fb58b7fe640 1 -- 172.21.15.64:0/3632795973 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11921 (secure 0 0 0) 0x7fb594199530 con 0x7fb594102250 2024-08-19T01:33:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:23.532 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":25,"btime":"2024-08-19T01:33:12:555328+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:33:12.505142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:33:23.533 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 25 2024-08-19T01:33:23.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb5800a2e10 msgr2=0x7fb5800a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:23.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb5800a2e10 0x7fb5800a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb59000bb60 tx=0x7fb59000b390 comp rx=0 tx=0).stop 2024-08-19T01:33:23.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb594102250 msgr2=0x7fb594198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb594102250 0x7fb594198710 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb584009230 tx=0x7fb584004270 comp rx=0 tx=0).stop 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 shutdown_connections 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb5800a2e10 0x7fb5800a52d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5941078b0 0x7fb59419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb594102250 0x7fb594198710 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 --2- 172.21.15.64:0/3632795973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5941018b0 0x7fb5941981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 >> 172.21.15.64:0/3632795973 conn(0x7fb5940fd5f0 msgr2=0x7fb5940fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 shutdown_connections 2024-08-19T01:33:23.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:23.533+0000 7fb59c484640 1 -- 172.21.15.64:0/3632795973 wait complete. 2024-08-19T01:33:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:23 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3632795973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:23 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3632795973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:23 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3632795973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:24.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:33:24.537 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:24.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:24 smithi064 ceph-mon[18515]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 60 op/s 2024-08-19T01:33:25.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 -- 172.21.15.64:0/2569245079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f01078b0 msgr2=0x7f21f0107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:25.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 --2- 172.21.15.64:0/2569245079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f01078b0 0x7f21f0107c90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f21e0009930 tx=0x7f21e002a420 comp rx=0 tx=0).stop 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 -- 172.21.15.64:0/2569245079 shutdown_connections 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 --2- 172.21.15.64:0/2569245079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f0102250 0x7f21f010e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 --2- 172.21.15.64:0/2569245079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21f01018b0 0x7f21f0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 --2- 172.21.15.64:0/2569245079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f01078b0 0x7f21f0107c90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 -- 172.21.15.64:0/2569245079 >> 172.21.15.64:0/2569245079 conn(0x7f21f00fd5f0 msgr2=0x7f21f00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 -- 172.21.15.64:0/2569245079 shutdown_connections 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 -- 172.21.15.64:0/2569245079 wait complete. 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.004+0000 7f21f5530640 1 Processor -- start 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21f5530640 1 -- start start 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21f5530640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21f01018b0 0x7f21f0074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21f5530640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f0102250 0x7f21f0074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21f5530640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f01078b0 0x7f21f0074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21f5530640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f21f0111d20 con 0x7f21f0102250 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21f5530640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f21f0111ba0 con 0x7f21f01078b0 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21f5530640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21f0111ea0 con 0x7f21f01018b0 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f0102250 0x7f21f0074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f0102250 0x7f21f0074760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50918/0 (socket says 172.21.15.64:50918) 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 -- 172.21.15.64:0/2538683589 learned_addr learned my addr 172.21.15.64:0/2538683589 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21eeffd640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21f01018b0 0x7f21f0074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ef7fe640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f01078b0 0x7f21f0074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 -- 172.21.15.64:0/2538683589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21f01018b0 msgr2=0x7f21f0074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21f01018b0 0x7f21f0074220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 -- 172.21.15.64:0/2538683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f01078b0 msgr2=0x7f21f0074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f01078b0 0x7f21f0074ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.005+0000 7f21ee7fc640 1 -- 172.21.15.64:0/2538683589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f01a6680 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.006+0000 7f21ee7fc640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f0102250 0x7f21f0074760 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f21e400b4a0 tx=0x7f21e400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.006+0000 7f21d7fff640 1 -- 172.21.15.64:0/2538683589 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4014070 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.006+0000 7f21d7fff640 1 -- 172.21.15.64:0/2538683589 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21e4010af0 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.006+0000 7f21d7fff640 1 -- 172.21.15.64:0/2538683589 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017d10 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.006+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f01a6970 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.006+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f01a6eb0 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.007+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f0101d10 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.008+0000 7f21d7fff640 1 -- 172.21.15.64:0/2538683589 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f21e4017070 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.009+0000 7f21d7fff640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f21d80a2e90 0x7f21d80a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.009+0000 7f21eeffd640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f21d80a2e90 0x7f21d80a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.009+0000 7f21d7fff640 1 -- 172.21.15.64:0/2538683589 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f21e40ced00 con 0x7f21f0102250 2024-08-19T01:33:25.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.009+0000 7f21eeffd640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f21d80a2e90 0x7f21d80a5350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f21e0006010 tx=0x7f21e0000bc0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:25.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.013+0000 7f21d7fff640 1 -- 172.21.15.64:0/2538683589 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f21e409b430 con 0x7f21f0102250 2024-08-19T01:33:25.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:24 smithi123 ceph-mon[21829]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 60 op/s 2024-08-19T01:33:25.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:24 smithi161 ceph-mon[21757]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 60 op/s 2024-08-19T01:33:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.223+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21f01a7140 con 0x7f21f0102250 2024-08-19T01:33:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.224+0000 7f21d7fff640 1 -- 172.21.15.64:0/2538683589 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11921 (secure 0 0 0) 0x7f21e409ad30 con 0x7f21f0102250 2024-08-19T01:33:25.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:25.226 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":25,"btime":"2024-08-19T01:33:12:555328+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:33:12.505142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:33:25.226 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 25 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f21d80a2e90 msgr2=0x7f21d80a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f21d80a2e90 0x7f21d80a5350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f21e0006010 tx=0x7f21e0000bc0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f0102250 msgr2=0x7f21f0074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f0102250 0x7f21f0074760 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f21e400b4a0 tx=0x7f21e400b960 comp rx=0 tx=0).stop 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 shutdown_connections 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f21d80a2e90 0x7f21d80a5350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21f01078b0 0x7f21f0074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f21f0102250 0x7f21f0074760 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 --2- 172.21.15.64:0/2538683589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21f01018b0 0x7f21f0074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 >> 172.21.15.64:0/2538683589 conn(0x7f21f00fd5f0 msgr2=0x7f21f00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 shutdown_connections 2024-08-19T01:33:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:25.226+0000 7f21f5530640 1 -- 172.21.15.64:0/2538683589 wait complete. 2024-08-19T01:33:25.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:33:25.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:25 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2538683589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:25 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2538683589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:26.172 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:25 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2538683589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.719+0000 7fda2524d640 1 -- 172.21.15.64:0/705009296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 msgr2=0x7fda2010d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.719+0000 7fda2524d640 1 --2- 172.21.15.64:0/705009296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda2010d660 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fda18009cd0 tx=0x7fda1802a8e0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 -- 172.21.15.64:0/705009296 shutdown_connections 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 --2- 172.21.15.64:0/705009296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda2010d660 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 --2- 172.21.15.64:0/705009296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda20100790 0x7fda20100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 --2- 172.21.15.64:0/705009296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda20106790 0x7fda20106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 -- 172.21.15.64:0/705009296 >> 172.21.15.64:0/705009296 conn(0x7fda200fc4b0 msgr2=0x7fda200fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 -- 172.21.15.64:0/705009296 shutdown_connections 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 -- 172.21.15.64:0/705009296 wait complete. 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.720+0000 7fda2524d640 1 Processor -- start 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda2524d640 1 -- start start 2024-08-19T01:33:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda2524d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda20100790 0x7fda200fcae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda2524d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda200fd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda2524d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda20106790 0x7fda200fd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda2524d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fda20110c20 con 0x7fda20101130 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda2524d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda20110aa0 con 0x7fda20100790 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda2524d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda20110da0 con 0x7fda20106790 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda200fd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda200fd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50930/0 (socket says 172.21.15.64:50930) 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 -- 172.21.15.64:0/3695693987 learned_addr learned my addr 172.21.15.64:0/3695693987 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1ed9d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda20100790 0x7fda200fcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1f59e640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda20106790 0x7fda200fd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 -- 172.21.15.64:0/3695693987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda20106790 msgr2=0x7fda200fd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda20106790 0x7fda200fd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 -- 172.21.15.64:0/3695693987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda20100790 msgr2=0x7fda200fcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda20100790 0x7fda200fcae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 -- 172.21.15.64:0/3695693987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda201a6660 con 0x7fda20101130 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1ed9d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda20100790 0x7fda200fcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1f59e640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda20106790 0x7fda200fd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.721+0000 7fda1e59c640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda200fd020 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fda0800d770 tx=0x7fda0800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.722+0000 7fda17fff640 1 -- 172.21.15.64:0/3695693987 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08017070 con 0x7fda20101130 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.722+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda201a6950 con 0x7fda20101130 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.722+0000 7fda17fff640 1 -- 172.21.15.64:0/3695693987 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda08005500 con 0x7fda20101130 2024-08-19T01:33:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.722+0000 7fda17fff640 1 -- 172.21.15.64:0/3695693987 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08004e00 con 0x7fda20101130 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.722+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda201a6e40 con 0x7fda20101130 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.723+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda20100bf0 con 0x7fda20101130 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.724+0000 7fda17fff640 1 -- 172.21.15.64:0/3695693987 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fda08004fa0 con 0x7fda20101130 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.725+0000 7fda17fff640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda0c0a2e10 0x7fda0c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.725+0000 7fda1ed9d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda0c0a2e10 0x7fda0c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.725+0000 7fda17fff640 1 -- 172.21.15.64:0/3695693987 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fda080cfa10 con 0x7fda20101130 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.726+0000 7fda1ed9d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda0c0a2e10 0x7fda0c0a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda10006fd0 tx=0x7fda10008040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.728+0000 7fda17fff640 1 -- 172.21.15.64:0/3695693987 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fda0809c030 con 0x7fda20101130 2024-08-19T01:33:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.906+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda20109d30 con 0x7fda20101130 2024-08-19T01:33:26.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:26 smithi064 ceph-mon[18515]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-08-19T01:33:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.908+0000 7fda17fff640 1 -- 172.21.15.64:0/3695693987 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25208 (secure 0 0 0) 0x7fda0809b930 con 0x7fda20101130 2024-08-19T01:33:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:26.910 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":94,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:33:13.815275+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/2663741075":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1810737759":"2024-08-20T01:33:13.815234+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1532949280":"2024-08-20T01:33:13.815234+0000","172.21.15.64:6800/2663741075":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1697666313":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1603406100":"2024-08-20T01:33:13.815234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:33:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda0c0a2e10 msgr2=0x7fda0c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda0c0a2e10 0x7fda0c0a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda10006fd0 tx=0x7fda10008040 comp rx=0 tx=0).stop 2024-08-19T01:33:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 msgr2=0x7fda200fd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda200fd020 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fda0800d770 tx=0x7fda0800dc30 comp rx=0 tx=0).stop 2024-08-19T01:33:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 shutdown_connections 2024-08-19T01:33:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda0c0a2e10 0x7fda0c0a52d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda20106790 0x7fda200fd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda20101130 0x7fda200fd020 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 --2- 172.21.15.64:0/3695693987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda20100790 0x7fda200fcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.910+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 >> 172.21.15.64:0/3695693987 conn(0x7fda200fc4b0 msgr2=0x7fda2006a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.911+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 shutdown_connections 2024-08-19T01:33:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:26.911+0000 7fda2524d640 1 -- 172.21.15.64:0/3695693987 wait complete. 2024-08-19T01:33:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:26 smithi123 ceph-mon[21829]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-08-19T01:33:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:26 smithi161 ceph-mon[21757]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-08-19T01:33:27.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-08-19T01:33:27.859 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:28.147 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:27 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3695693987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:33:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:27 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3695693987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:33:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:27 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3695693987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:33:28.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:28 smithi064 ceph-mon[18515]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-08-19T01:33:29.127 INFO:teuthology.orchestra.run.smithi064.stdout:{"run_dir":"/var/run/ceph"} 2024-08-19T01:33:29.128 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-08-19T01:33:29.128 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 1664 2024-08-19T01:33:29.128 DEBUG:tasks.quiescer:Yielding 2024-08-19T01:33:29.128 INFO:teuthology.run_tasks:Running task workunit... 2024-08-19T01:33:29.144 INFO:tasks.workunit:Pulling workunits from ref 78215f5a1cf5eaf321d506a66539942e12851f34 2024-08-19T01:33:29.144 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-19T01:33:29.145 INFO:tasks.workunit:timeout=3h 2024-08-19T01:33:29.145 INFO:tasks.workunit:cleanup=True 2024-08-19T01:33:29.145 DEBUG:teuthology.orchestra.run.smithi064:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:29.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:33:29.163 INFO:teuthology.orchestra.run.smithi064.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:29.163 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-08-19T01:33:29.164 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2024-08-19T01:33:29.164 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T01:33:29.164 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-08-19T01:33:29.164 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-08-19 01:31:42.366770020 +0000 2024-08-19T01:33:29.164 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-08-19 01:31:42.366770020 +0000 2024-08-19T01:33:29.164 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-08-19 01:33:11.786011390 +0000 2024-08-19T01:33:29.164 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-08-19 01:31:42.366770020 +0000 2024-08-19T01:33:29.164 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-08-19T01:33:29.164 DEBUG:teuthology.orchestra.run.smithi064:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-08-19T01:33:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:28 smithi123 ceph-mon[21829]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-08-19T01:33:29.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:28 smithi161 ceph-mon[21757]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-08-19T01:33:29.237 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 78215f5a1cf5eaf321d506a66539942e12851f34 2024-08-19T01:33:29.293 INFO:tasks.workunit.client.0.smithi064.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-19T01:33:29.413 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:29.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 -- 172.21.15.64:0/1885622065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41067c0 msgr2=0x7fdad4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:29.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 --2- 172.21.15.64:0/1885622065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41067c0 0x7fdad4106ba0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009930 tx=0x7fdac802a420 comp rx=0 tx=0).stop 2024-08-19T01:33:29.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 -- 172.21.15.64:0/1885622065 shutdown_connections 2024-08-19T01:33:29.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 --2- 172.21.15.64:0/1885622065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdad4101160 0x7fdad410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:29.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 --2- 172.21.15.64:0/1885622065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad41007c0 0x7fdad4100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:29.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 --2- 172.21.15.64:0/1885622065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41067c0 0x7fdad4106ba0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:29.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 -- 172.21.15.64:0/1885622065 >> 172.21.15.64:0/1885622065 conn(0x7fdad40fc500 msgr2=0x7fdad40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 -- 172.21.15.64:0/1885622065 shutdown_connections 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.901+0000 7fdadac1d640 1 -- 172.21.15.64:0/1885622065 wait complete. 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 Processor -- start 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 -- start start 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41007c0 0x7fdad40fcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad4101160 0x7fdad40fd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdad41067c0 0x7fdad40fd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdad4110c00 con 0x7fdad41007c0 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdad4110a80 con 0x7fdad41067c0 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdadac1d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdad4110d80 con 0x7fdad4101160 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdad89b9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41007c0 0x7fdad40fcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdad89b9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41007c0 0x7fdad40fcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55958/0 (socket says 172.21.15.64:55958) 2024-08-19T01:33:29.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.902+0000 7fdad89b9640 1 -- 172.21.15.64:0/3068232195 learned_addr learned my addr 172.21.15.64:0/3068232195 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad3fff640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad4101160 0x7fdad40fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad91ba640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdad41067c0 0x7fdad40fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad89b9640 1 -- 172.21.15.64:0/3068232195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad4101160 msgr2=0x7fdad40fd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad89b9640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad4101160 0x7fdad40fd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad89b9640 1 -- 172.21.15.64:0/3068232195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdad41067c0 msgr2=0x7fdad40fd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad89b9640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdad41067c0 0x7fdad40fd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad89b9640 1 -- 172.21.15.64:0/3068232195 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad41a66e0 con 0x7fdad41007c0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad89b9640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41007c0 0x7fdad40fcaf0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009230 tx=0x7fdac8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad1ffb640 1 -- 172.21.15.64:0/3068232195 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac803f030 con 0x7fdad41007c0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad1ffb640 1 -- 172.21.15.64:0/3068232195 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac8003ad0 con 0x7fdad41007c0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad41a6970 con 0x7fdad41007c0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdad1ffb640 1 -- 172.21.15.64:0/3068232195 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8035070 con 0x7fdad41007c0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.903+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad41a6ea0 con 0x7fdad41007c0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.905+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad4100c20 con 0x7fdad41007c0 2024-08-19T01:33:29.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.905+0000 7fdad1ffb640 1 -- 172.21.15.64:0/3068232195 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fdac8042030 con 0x7fdad41007c0 2024-08-19T01:33:29.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.906+0000 7fdad1ffb640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdac40a2e90 0x7fdac40a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:29.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.906+0000 7fdad3fff640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdac40a2e90 0x7fdac40a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:29.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.906+0000 7fdad1ffb640 1 -- 172.21.15.64:0/3068232195 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdac80efee0 con 0x7fdad41007c0 2024-08-19T01:33:29.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.907+0000 7fdad3fff640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdac40a2e90 0x7fdac40a5350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdac0005fd0 tx=0x7fdac0005850 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:29.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:29.911+0000 7fdad1ffb640 1 -- 172.21.15.64:0/3068232195 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fdac80bc5d0 con 0x7fdad41007c0 2024-08-19T01:33:30.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.129+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdad4109d40 con 0x7fdad41007c0 2024-08-19T01:33:30.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.129+0000 7fdad1ffb640 1 -- 172.21.15.64:0/3068232195 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11921 (secure 0 0 0) 0x7fdac80bbed0 con 0x7fdad41007c0 2024-08-19T01:33:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:30.131 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":25,"btime":"2024-08-19T01:33:12:555328+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:33:12.505142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:33:30.131 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 25 2024-08-19T01:33:30.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdac40a2e90 msgr2=0x7fdac40a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:30.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdac40a2e90 0x7fdac40a5350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdac0005fd0 tx=0x7fdac0005850 comp rx=0 tx=0).stop 2024-08-19T01:33:30.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41007c0 msgr2=0x7fdad40fcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:30.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41007c0 0x7fdad40fcaf0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009230 tx=0x7fdac8004270 comp rx=0 tx=0).stop 2024-08-19T01:33:30.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 shutdown_connections 2024-08-19T01:33:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdac40a2e90 0x7fdac40a5350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdad41067c0 0x7fdad40fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad4101160 0x7fdad40fd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 --2- 172.21.15.64:0/3068232195 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdad41007c0 0x7fdad40fcaf0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 >> 172.21.15.64:0/3068232195 conn(0x7fdad40fc500 msgr2=0x7fdad406a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 shutdown_connections 2024-08-19T01:33:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:30.132+0000 7fdadac1d640 1 -- 172.21.15.64:0/3068232195 wait complete. 2024-08-19T01:33:30.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-08-19T01:30:52.282253+0000', 'modified': '2024-08-19T01:33:12.505142+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24476, 'mds_1': 24452, 'mds_2': 24436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24415': {'gid': 24415, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/3530643584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3530643584}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 3530643584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6837/1326350329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 1326350329}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1326350329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6835/2709386676', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 2709386676}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2709386676}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.161:6839/2198702470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2198702470}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2198702470}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.123:6837/3425782983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6836', 'nonce': 3425782983}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 3425782983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/2281975354', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 2281975354}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 2281975354}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24476, 'qdb_cluster': [24436, 24452, 24476]} 2024-08-19T01:33:30.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-08-19T01:33:30.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.i status 2024-08-19T01:33:31.102 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:30 smithi123 ceph-mon[21829]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-08-19T01:33:31.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:30 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3068232195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:30 smithi161 ceph-mon[21757]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-08-19T01:33:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:30 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3068232195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:31.225 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:30 smithi064 ceph-mon[18515]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-08-19T01:33:31.225 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:30 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3068232195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1287596402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 msgr2=0x7f61e8100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1287596402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8100a80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009930 tx=0x7f61dc02a420 comp rx=0 tx=0).stop 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1287596402 shutdown_connections 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1287596402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61e8100fc0 0x7f61e810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1287596402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8100a80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1287596402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61e8106690 0x7f61e8106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1287596402 >> 172.21.15.64:0/1287596402 conn(0x7f61e80fc340 msgr2=0x7f61e80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:31.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1287596402 shutdown_connections 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.628+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1287596402 wait complete. 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 Processor -- start 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 -- start start 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61e8100fc0 0x7f61e81992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61e8106690 0x7f61e81a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f61e81109f0 con 0x7f61e8100600 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8110870 con 0x7f61e8106690 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61ecb9f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8110b70 con 0x7f61e8100fc0 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61e77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61e77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55976/0 (socket says 172.21.15.64:55976) 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61e77fe640 1 -- 172.21.15.64:0/1314047424 learned_addr learned my addr 172.21.15.64:0/1314047424 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:31.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61e8100fc0 0x7f61e81992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.629+0000 7f61e7fff640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61e8106690 0x7f61e81a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e77fe640 1 -- 172.21.15.64:0/1314047424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61e8100fc0 msgr2=0x7f61e81992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e77fe640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61e8100fc0 0x7f61e81992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e77fe640 1 -- 172.21.15.64:0/1314047424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61e8106690 msgr2=0x7f61e81a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e77fe640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61e8106690 0x7f61e81a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e77fe640 1 -- 172.21.15.64:0/1314047424 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e81a4340 con 0x7f61e8100600 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61e8100fc0 0x7f61e81992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e77fe640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8198d80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f61d800d950 tx=0x7f61d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e4ff9640 1 -- 172.21.15.64:0/1314047424 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d8017070 con 0x7f61e8100600 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e81a4510 con 0x7f61e8100600 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e4ff9640 1 -- 172.21.15.64:0/1314047424 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61d8009c30 con 0x7f61e8100600 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61e4ff9640 1 -- 172.21.15.64:0/1314047424 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d800f6b0 con 0x7f61e8100600 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.630+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e81a49d0 con 0x7f61e8100600 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e4ff9640 1 -- 172.21.15.64:0/1314047424 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f61d80109d0 con 0x7f61e8100600 2024-08-19T01:33:31.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 Processor -- start 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 -- start start 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c51b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c520d0 0x563089c5c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c5c990 0x563089c5ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563089bb6710 con 0x563089c517b0 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563089b4e770 con 0x563089c5c990 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61f01ff740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563089bb26d0 con 0x563089c520d0 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c51b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c51b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55984/0 (socket says 172.21.15.64:55984) 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 -- 172.21.15.64:0/2361971725 learned_addr learned my addr 172.21.15.64:0/2361971725 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e77fe640 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c520d0 0x563089c5c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 -- 172.21.15.64:0/2361971725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c520d0 msgr2=0x563089c5c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c5c990 0x563089c5ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c520d0 0x563089c5c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 -- 172.21.15.64:0/2361971725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c5c990 msgr2=0x563089c5ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e4ff9640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f61d00a2e10 0x7f61d00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c5c990 0x563089c5ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e7fff640 1 -- 172.21.15.64:0/2361971725 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563089b553e0 con 0x563089c517b0 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.633+0000 7f61e77fe640 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c520d0 0x563089c5c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.634+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c5c990 0x563089c5ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.634+0000 7f61e4ff9640 1 -- 172.21.15.64:0/1314047424 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f61d80cf760 con 0x7f61e8100600 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.634+0000 7f61e7fff640 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c51b90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009ee0 tx=0x7f61e0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c15d3da89ed5f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.634+0000 7f61e7fff640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f61d00a2e10 0x7f61d00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.634+0000 7f61cdffb640 1 -- 172.21.15.64:0/2361971725 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e0021070 con 0x563089c517b0 2024-08-19T01:33:31.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.634+0000 7f61cdffb640 1 -- 172.21.15.64:0/2361971725 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e00113e0 con 0x563089c517b0 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.634+0000 7f61cdffb640 1 -- 172.21.15.64:0/2361971725 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e0016070 con 0x563089c517b0 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 -- 172.21.15.64:0/2361971725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 msgr2=0x563089c51b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61e7fff640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f61d00a2e10 0x7f61d00a52d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f61e001e000 tx=0x7f61e0021000 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c51b90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009ee0 tx=0x7f61e0010740 comp rx=0 tx=0).stop 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 -- 172.21.15.64:0/2361971725 shutdown_connections 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c5c990 0x563089c5ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c520d0 0x563089c5c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 --2- 172.21.15.64:0/2361971725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c51b90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 -- 172.21.15.64:0/2361971725 >> 172.21.15.64:0/2361971725 conn(0x563089c4dd40 msgr2=0x563089c4e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 -- 172.21.15.64:0/2361971725 shutdown_connections 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 -- 172.21.15.64:0/2361971725 wait complete. 2024-08-19T01:33:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.635+0000 7f61f01ff740 1 -- start start 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 1 Processor -- start 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c517b0 0x563089c59800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c59d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c5c990 0x563089c5a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563089c5f560 con 0x563089c5c990 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563089b71970 con 0x563089c520d0 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563089c5f6e0 con 0x563089c517b0 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61f01ff740 10 client.0 _ll_register_callbacks cb 0x5630898c57b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c5c990 0x563089c5a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c59d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c5c990 0x563089c5a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55986/0 (socket says 172.21.15.64:55986) 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c59d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:42502/0 (socket says 172.21.15.64:42502) 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 -- 172.21.15.64:0/4109137510 learned_addr learned my addr 172.21.15.64:0/4109137510 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e77fe640 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c517b0 0x563089c59800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 -- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c517b0 msgr2=0x563089c59800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c517b0 0x563089c59800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 -- 172.21.15.64:0/4109137510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 msgr2=0x563089c59d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c59d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 -- 172.21.15.64:0/4109137510 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630899282b0 con 0x563089c5c990 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61e7fff640 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c5c990 0x563089c5a280 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f61e0011e30 tx=0x7f61e00129a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795f3199cbe7e942 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.636+0000 7f61c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61c6ffd640 1 -- 172.21.15.64:0/4109137510 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e0016040 con 0x563089c5c990 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61e00113e0 con 0x563089c5c990 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61c6ffd640 1 -- 172.21.15.64:0/4109137510 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f61e00049e0 con 0x563089c5c990 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61c6ffd640 1 -- 172.21.15.64:0/4109137510 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f61e0004be0 con 0x563089c5c990 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 10 client.15423 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:31.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563089c6b060 con 0x563089c5c990 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61c6ffd640 1 -- 172.21.15.64:0/4109137510 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e00113e0 con 0x563089c5c990 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61c6ffd640 1 -- 172.21.15.64:0/4109137510 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f61e002e030 con 0x563089c5c990 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 10 client.15423 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 10 client.15423 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 20 client.15423 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089c6d480 0x563089c6e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 4 client.15423 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.637+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 --> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563089bc3c80 con 0x563089c6d480 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.638+0000 7f61e77fe640 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089c6d480 0x563089c6e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.638+0000 7f61c6ffd640 10 client.15423 ms_handle_connect on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:31.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.638+0000 7f61e77fe640 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089c6d480 0x563089c6e8f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d2e3edbb98afb3 server_cookie=ee416300d493e7d1 in_seq=0 out_seq=0 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61c6ffd640 1 -- 172.21.15.64:0/4109137510 <== mds.2 v2:172.21.15.161:6838/2198702470 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563089bc3c80 con 0x563089c6d480 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61c6ffd640 10 client.15423 handle_command_reply: tid=0 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 client.15423 shutdown 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c5c990 msgr2=0x563089c5a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c5c990 0x563089c5a280 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f61e0011e30 tx=0x7f61e00129a0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 shutdown_connections 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089c6d480 0x563089c6e8f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c5c990 0x563089c5a280 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61c6ffd640 0 client.15423 ms_handle_reset on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c59d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 --2- 172.21.15.64:0/4109137510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c517b0 0x563089c59800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.639+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 >> 172.21.15.64:0/4109137510 conn(0x563089c4dd40 msgr2=0x563089c53a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.640+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 shutdown_connections 2024-08-19T01:33:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.640+0000 7f61f01ff740 1 -- 172.21.15.64:0/4109137510 wait complete. 2024-08-19T01:33:31.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.640+0000 7f61f01ff740 20 client.15423 trim_cache size 0 max 16384 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 Processor -- start 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 -- start start 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c4d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c56a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089c56fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563089c6cc30 con 0x563089c517b0 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563089c6d300 con 0x563089c520d0 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61f01ff740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563089c6d9d0 con 0x563089c5c990 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c4d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c4d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55992/0 (socket says 172.21.15.64:55992) 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 -- 172.21.15.64:0/341762481 learned_addr learned my addr 172.21.15.64:0/341762481 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c56a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e7fff640 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089c56fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 -- 172.21.15.64:0/341762481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 msgr2=0x563089c56fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089c56fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 -- 172.21.15.64:0/341762481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 msgr2=0x563089c56a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c56a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e77fe640 1 -- 172.21.15.64:0/341762481 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563089c5c2b0 con 0x563089c517b0 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c56a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.658+0000 7f61e7fff640 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089c56fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61e77fe640 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c4d5f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f61d8010fd0 tx=0x7f61d80d1c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0d06811c1893bf server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61cd7fa640 1 -- 172.21.15.64:0/341762481 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d8020040 con 0x563089c517b0 2024-08-19T01:33:31.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61cd7fa640 1 -- 172.21.15.64:0/341762481 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61d809b680 con 0x563089c517b0 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61cd7fa640 1 -- 172.21.15.64:0/341762481 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d80a3e60 con 0x563089c517b0 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 -- 172.21.15.64:0/341762481 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 msgr2=0x563089c4d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c4d5f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f61d8010fd0 tx=0x7f61d80d1c40 comp rx=0 tx=0).stop 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 -- 172.21.15.64:0/341762481 shutdown_connections 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089c56fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c520d0 0x563089c56a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 --2- 172.21.15.64:0/341762481 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c517b0 0x563089c4d5f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 -- 172.21.15.64:0/341762481 >> 172.21.15.64:0/341762481 conn(0x563089c4dd40 msgr2=0x563089c54ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 -- 172.21.15.64:0/341762481 shutdown_connections 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.659+0000 7f61f01ff740 1 -- 172.21.15.64:0/341762481 wait complete. 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 -- start start 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 Processor -- start 2024-08-19T01:33:31.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c517b0 0x563089c72250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c520d0 0x563089c72790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089ccfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563089c6e770 con 0x563089c520d0 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x563089c6e0a0 con 0x563089c517b0 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563089c6ee40 con 0x563089c5c990 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61f01ff740 10 client.0 _ll_register_callbacks cb 0x7f61e001f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61e6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c520d0 0x563089c72790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61e6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c520d0 0x563089c72790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55998/0 (socket says 172.21.15.64:55998) 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61e6ffd640 1 -- 172.21.15.64:0/126633421 learned_addr learned my addr 172.21.15.64:0/126633421 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.660+0000 7f61e77fe640 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c517b0 0x563089c72250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61e7fff640 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089ccfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61e6ffd640 1 -- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 msgr2=0x563089ccfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089ccfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61e6ffd640 1 -- 172.21.15.64:0/126633421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c517b0 msgr2=0x563089c72250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c517b0 0x563089c72250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61e6ffd640 1 -- 172.21.15.64:0/126633421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563089cd0200 con 0x563089c520d0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61e6ffd640 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c520d0 0x563089c72790 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f61dc02a940 tx=0x7f61dc031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f93c082fc3b80a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61c6ffd640 1 -- 172.21.15.64:0/126633421 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc035040 con 0x563089c520d0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61c6ffd640 1 -- 172.21.15.64:0/126633421 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f61dc004240 con 0x563089c520d0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61d809b680 con 0x563089c520d0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61c6ffd640 1 -- 172.21.15.64:0/126633421 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f61dc04e050 con 0x563089c520d0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61f01ff740 10 client.15429 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.661+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563089cd7510 con 0x563089c520d0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61c6ffd640 1 -- 172.21.15.64:0/126633421 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61d809b680 con 0x563089c520d0 2024-08-19T01:33:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61c6ffd640 1 -- 172.21.15.64:0/126633421 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f61dc004bf0 con 0x563089c520d0 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61f01ff740 10 client.15429 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61f01ff740 10 client.15429 resolve_mds: resolved name 'i' to daemon mds.i 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61f01ff740 20 client.15429 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61f01ff740 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089cdc120 0x563089cdae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61f01ff740 4 client.15429 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 --> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] -- command(tid 0: {"prefix": "status"}) -- 0x563089bbe190 con 0x563089cdc120 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.662+0000 7f61e77fe640 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089cdc120 0x563089cdae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61e77fe640 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089cdc120 0x563089cdae30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2230f12d6d906460 server_cookie=d1a1bbf95bf46a24 in_seq=0 out_seq=0 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61c6ffd640 10 client.15429 ms_handle_connect on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61c6ffd640 1 -- 172.21.15.64:0/126633421 <== mds.2 v2:172.21.15.161:6838/2198702470 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563089bbe190 con 0x563089cdc120 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61c6ffd640 10 client.15429 handle_command_reply: tid=0 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 client.15429 shutdown 2024-08-19T01:33:31.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c520d0 msgr2=0x563089c72790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c520d0 0x563089c72790 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f61dc02a940 tx=0x7f61dc031920 comp rx=0 tx=0).stop 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 shutdown_connections 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:6838/2198702470,v1:172.21.15.161:6839/2198702470] conn(0x563089cdc120 0x563089cdae30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563089c5c990 0x563089ccfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563089c520d0 0x563089c72790 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 --2- 172.21.15.64:0/126633421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563089c517b0 0x563089c72250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 >> 172.21.15.64:0/126633421 conn(0x563089c4dd40 msgr2=0x563089c53200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61c6ffd640 0 client.15429 ms_handle_reset on v2:172.21.15.161:6838/2198702470 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 shutdown_connections 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.663+0000 7f61f01ff740 1 -- 172.21.15.64:0/126633421 wait complete. 2024-08-19T01:33:31.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61f01ff740 20 client.15429 trim_cache size 0 max 16384 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24436, 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 118.920393434, 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 25, 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 94, 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 93, 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 179.12797183999999 2024-08-19T01:33:31.675 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f61d00a2e10 msgr2=0x7f61d00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f61d00a2e10 0x7f61d00a52d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f61e001e000 tx=0x7f61e0021000 comp rx=0 tx=0).stop 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 msgr2=0x7f61e8198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8198d80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f61d800d950 tx=0x7f61d800de10 comp rx=0 tx=0).stop 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 shutdown_connections 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f61d00a2e10 0x7f61d00a52d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61e8106690 0x7f61e81a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61e8100fc0 0x7f61e81992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 --2- 172.21.15.64:0/1314047424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f61e8100600 0x7f61e8198d80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:31.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 >> 172.21.15.64:0/1314047424 conn(0x7f61e80fc340 msgr2=0x7f61e80fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:31.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 shutdown_connections 2024-08-19T01:33:31.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:31.664+0000 7f61ecb9f640 1 -- 172.21.15.64:0/1314047424 wait complete. 2024-08-19T01:33:32.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.h status 2024-08-19T01:33:32.615 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:33.013 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:32 smithi064 ceph-mon[18515]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-08-19T01:33:33.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 -- 172.21.15.64:0/2728425676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 msgr2=0x7faa18102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 --2- 172.21.15.64:0/2728425676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa18102e50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7faa08009900 tx=0x7faa0802a480 comp rx=0 tx=0).stop 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 -- 172.21.15.64:0/2728425676 shutdown_connections 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 --2- 172.21.15.64:0/2728425676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa1810bea0 0x7faa1810e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 --2- 172.21.15.64:0/2728425676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa18103390 0x7faa1810b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 --2- 172.21.15.64:0/2728425676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa18102e50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 -- 172.21.15.64:0/2728425676 >> 172.21.15.64:0/2728425676 conn(0x7faa180fc4d0 msgr2=0x7faa180fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 -- 172.21.15.64:0/2728425676 shutdown_connections 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.100+0000 7faa1db02640 1 -- 172.21.15.64:0/2728425676 wait complete. 2024-08-19T01:33:33.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 Processor -- start 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 -- start start 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa181981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa18103390 0x7faa181986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1810bea0 0x7faa1819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa18111ca0 con 0x7faa18102a70 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa18111b20 con 0x7faa1810bea0 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa1db02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faa18111e20 con 0x7faa18103390 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa181981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa181981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56018/0 (socket says 172.21.15.64:56018) 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa177fe640 1 -- 172.21.15.64:0/999759741 learned_addr learned my addr 172.21.15.64:0/999759741 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa16ffd640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa18103390 0x7faa181986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.101+0000 7faa17fff640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1810bea0 0x7faa1819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa177fe640 1 -- 172.21.15.64:0/999759741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa18103390 msgr2=0x7faa181986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa177fe640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa18103390 0x7faa181986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa177fe640 1 -- 172.21.15.64:0/999759741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1810bea0 msgr2=0x7faa1819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa177fe640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1810bea0 0x7faa1819ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa177fe640 1 -- 172.21.15.64:0/999759741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1819d150 con 0x7faa18102a70 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa17fff640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1810bea0 0x7faa1819ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa177fe640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa181981a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faa08005ab0 tx=0x7faa080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa14ff9640 1 -- 172.21.15.64:0/999759741 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0803f030 con 0x7faa18102a70 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa14ff9640 1 -- 172.21.15.64:0/999759741 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa08003ce0 con 0x7faa18102a70 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa1819d3e0 con 0x7faa18102a70 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa14ff9640 1 -- 172.21.15.64:0/999759741 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08035070 con 0x7faa18102a70 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.102+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa181a4d20 con 0x7faa18102a70 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 Processor -- start 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa14ff9640 1 -- 172.21.15.64:0/999759741 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7faa08042030 con 0x7faa18102a70 2024-08-19T01:33:33.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 -- start start 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afb38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afb3e30 0x558e4afc26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afc2c20 0x558e4afc5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558e4af18710 con 0x558e4afb3510 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558e4aeb0770 con 0x558e4afb3e30 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa1ff00740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558e4af146d0 con 0x558e4afc2c20 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa17fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afb38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa17fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afb38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56024/0 (socket says 172.21.15.64:56024) 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.105+0000 7faa17fff640 1 -- 172.21.15.64:0/3157671528 learned_addr learned my addr 172.21.15.64:0/3157671528 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa16ffd640 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afc2c20 0x558e4afc5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa177fe640 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afb3e30 0x558e4afc26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa17fff640 1 -- 172.21.15.64:0/3157671528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afc2c20 msgr2=0x558e4afc5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa17fff640 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afc2c20 0x558e4afc5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa17fff640 1 -- 172.21.15.64:0/3157671528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afb3e30 msgr2=0x558e4afc26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa17fff640 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afb3e30 0x558e4afc26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa17fff640 1 -- 172.21.15.64:0/3157671528 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e4aeb73e0 con 0x558e4afb3510 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa14ff9640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faa000a2e10 0x7faa000a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa177fe640 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afb3e30 0x558e4afc26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa177fe640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faa000a2e10 0x7faa000a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa17fff640 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afb38f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7faa10008d50 tx=0x7faa1000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c4d2921aa75125 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7faa14ff9640 1 -- 172.21.15.64:0/999759741 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7faa080f02b0 con 0x7faa18102a70 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7fa9fdffb640 1 -- 172.21.15.64:0/3157671528 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1000f830 con 0x558e4afb3510 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7fa9fdffb640 1 -- 172.21.15.64:0/3157671528 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1000de20 con 0x558e4afb3510 2024-08-19T01:33:33.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.106+0000 7fa9fdffb640 1 -- 172.21.15.64:0/3157671528 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1000e3b0 con 0x558e4afb3510 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 -- 172.21.15.64:0/3157671528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 msgr2=0x558e4afb38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa177fe640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faa000a2e10 0x7faa000a52d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa080ed530 tx=0x7faa08047000 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afb38f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7faa10008d50 tx=0x7faa1000d710 comp rx=0 tx=0).stop 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 -- 172.21.15.64:0/3157671528 shutdown_connections 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afc2c20 0x558e4afc5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afb3e30 0x558e4afc26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3157671528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afb38f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 -- 172.21.15.64:0/3157671528 >> 172.21.15.64:0/3157671528 conn(0x558e4afafbc0 msgr2=0x558e4afaffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 -- 172.21.15.64:0/3157671528 shutdown_connections 2024-08-19T01:33:33.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 -- 172.21.15.64:0/3157671528 wait complete. 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.107+0000 7faa1ff00740 1 -- start start 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 1 Processor -- start 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afbfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3e30 0x558e4afc0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afc0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558e4afc57f0 con 0x558e4afb3510 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558e4aed3970 con 0x558e4afc2c20 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558e4afc5970 con 0x558e4afb3e30 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa1ff00740 10 client.0 _ll_register_callbacks cb 0x558e4ac277b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afbfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afbfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56036/0 (socket says 172.21.15.64:56036) 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 -- 172.21.15.64:0/2219868713 learned_addr learned my addr 172.21.15.64:0/2219868713 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa16ffd640 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3e30 0x558e4afc0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 -- 172.21.15.64:0/2219868713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3e30 msgr2=0x558e4afc0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3e30 0x558e4afc0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 -- 172.21.15.64:0/2219868713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 msgr2=0x558e4afc0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afc0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 -- 172.21.15.64:0/2219868713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e4ac8a2b0 con 0x558e4afb3510 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa16ffd640 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3e30 0x558e4afc0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7faa17fff640 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afbfbc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7faa10008fa0 tx=0x7faa10008ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc2785509caa5b2 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.108+0000 7fa9f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/2219868713 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1001e070 con 0x558e4afb3510 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/2219868713 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7faa1000f630 con 0x558e4afb3510 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1000de20 con 0x558e4afb3510 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/2219868713 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faa1000f830 con 0x558e4afb3510 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 10 client.15441 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558e4afd1460 con 0x558e4afb3510 2024-08-19T01:33:33.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/2219868713 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1000de20 con 0x558e4afb3510 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/2219868713 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7faa10024c20 con 0x558e4afb3510 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 10 client.15441 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 10 client.15441 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 20 client.15441 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4afd36c0 0x558e4afd4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 4 client.15441 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.109+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 --> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e4af25c80 con 0x558e4afd36c0 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.110+0000 7faa16ffd640 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4afd36c0 0x558e4afd4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.110+0000 7fa9f6ffd640 10 client.15441 ms_handle_connect on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:33.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.110+0000 7faa16ffd640 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4afd36c0 0x558e4afd4b30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5a077fa1dca63c2 server_cookie=e8cf43f6dbb489b1 in_seq=0 out_seq=0 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/2219868713 <== mds.1 v2:172.21.15.123:6836/3425782983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e4af25c80 con 0x558e4afd36c0 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7fa9f6ffd640 10 client.15441 handle_command_reply: tid=0 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 client.15441 shutdown 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 msgr2=0x558e4afbfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afbfbc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7faa10008fa0 tx=0x7faa10008ac0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 shutdown_connections 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4afd36c0 0x558e4afd4b30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afc0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3e30 0x558e4afc0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 --2- 172.21.15.64:0/2219868713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3510 0x558e4afbfbc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7fa9f6ffd640 0 client.15441 ms_handle_reset on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:33.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 >> 172.21.15.64:0/2219868713 conn(0x558e4afafbc0 msgr2=0x558e4afb5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:33.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 shutdown_connections 2024-08-19T01:33:33.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 1 -- 172.21.15.64:0/2219868713 wait complete. 2024-08-19T01:33:33.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.111+0000 7faa1ff00740 20 client.15441 trim_cache size 0 max 16384 2024-08-19T01:33:33.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 Processor -- start 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 -- start start 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afcf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afbf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558e4afd2e70 con 0x558e4afb3e30 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558e4afd3540 con 0x558e4afc2c20 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa1ff00740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558e4afd3c10 con 0x558e4afb3510 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56046/0 (socket says 172.21.15.64:56046) 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.129+0000 7faa16ffd640 1 -- 172.21.15.64:0/3877024465 learned_addr learned my addr 172.21.15.64:0/3877024465 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:33.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7faa17fff640 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afcf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7faa16ffd640 1 -- 172.21.15.64:0/3877024465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 msgr2=0x558e4afcf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7faa16ffd640 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afcf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7faa16ffd640 1 -- 172.21.15.64:0/3877024465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 msgr2=0x558e4afbf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7faa16ffd640 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afbf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7faa16ffd640 1 -- 172.21.15.64:0/3877024465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e4afb3c60 con 0x558e4afb3e30 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7faa16ffd640 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd2ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x558e4afb0cb0 tx=0x7faa0c013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44dea5ffd79b32 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7fa9fd7fa640 1 -- 172.21.15.64:0/3877024465 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c01c460 con 0x558e4afb3e30 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7fa9fd7fa640 1 -- 172.21.15.64:0/3877024465 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa0c01cae0 con 0x558e4afb3e30 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.130+0000 7fa9fd7fa640 1 -- 172.21.15.64:0/3877024465 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c011a50 con 0x558e4afb3e30 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 -- 172.21.15.64:0/3877024465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 msgr2=0x558e4afd2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd2ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x558e4afb0cb0 tx=0x7faa0c013830 comp rx=0 tx=0).stop 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 -- 172.21.15.64:0/3877024465 shutdown_connections 2024-08-19T01:33:33.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afbf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd2ff0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 --2- 172.21.15.64:0/3877024465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afcf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 -- 172.21.15.64:0/3877024465 >> 172.21.15.64:0/3877024465 conn(0x558e4afafbc0 msgr2=0x558e4afaffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 -- 172.21.15.64:0/3877024465 shutdown_connections 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 -- 172.21.15.64:0/3877024465 wait complete. 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 -- start start 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 Processor -- start 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.131+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afdd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa1ff00740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afd6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa1ff00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558e4afd49b0 con 0x558e4afb3e30 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa1ff00740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558e4afd42e0 con 0x558e4afc2c20 2024-08-19T01:33:33.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa1ff00740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558e4afd5080 con 0x558e4afb3510 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa1ff00740 10 client.0 _ll_register_callbacks cb 0x7faa10008bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56048/0 (socket says 172.21.15.64:56048) 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 -- 172.21.15.64:0/1600120286 learned_addr learned my addr 172.21.15.64:0/1600120286 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa17fff640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afdd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa17fff640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afd6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 -- 172.21.15.64:0/1600120286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 msgr2=0x558e4afdd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afdd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 -- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 msgr2=0x558e4afd6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afd6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 -- 172.21.15.64:0/1600120286 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e4afd7650 con 0x558e4afb3e30 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa17fff640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afd6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:33.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa16ffd640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd68c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x558e4afc5030 tx=0x7faa0c01cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e409ab0ce0ac02 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7fa9f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/1600120286 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c004220 con 0x558e4afb3e30 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa0c01cae0 con 0x558e4afb3e30 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.132+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/1600120286 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7faa0c0043c0 con 0x558e4afb3e30 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/1600120286 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faa0c011440 con 0x558e4afb3e30 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 10 client.15447 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558e4b03de90 con 0x558e4afb3e30 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/1600120286 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa0c01cae0 con 0x558e4afb3e30 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/1600120286 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7faa0c02d070 con 0x558e4afb3e30 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 10 client.15447 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 10 client.15447 resolve_mds: resolved name 'h' to daemon mds.h 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 20 client.15447 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:33.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4b0427f0 0x558e4b041500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 4 client.15447 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 --> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] -- command(tid 0: {"prefix": "status"}) -- 0x558e4af20190 con 0x558e4b0427f0 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.133+0000 7faa17fff640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4b0427f0 0x558e4b041500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa17fff640 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4b0427f0 0x558e4b041500 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc7bd8a75bae2c0a server_cookie=626b764bf09a8185 in_seq=0 out_seq=0 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7fa9f6ffd640 10 client.15447 ms_handle_connect on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7fa9f6ffd640 1 -- 172.21.15.64:0/1600120286 <== mds.1 v2:172.21.15.123:6836/3425782983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558e4af20190 con 0x558e4b0427f0 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7fa9f6ffd640 10 client.15447 handle_command_reply: tid=0 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 client.15447 shutdown 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 msgr2=0x558e4afd68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd68c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x558e4afc5030 tx=0x7faa0c01cec0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 shutdown_connections 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:6836/3425782983,v1:172.21.15.123:6837/3425782983] conn(0x558e4b0427f0 0x558e4b041500 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558e4afc2c20 0x558e4afd6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e4afb3e30 0x558e4afd68c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7fa9f6ffd640 0 client.15447 ms_handle_reset on v2:172.21.15.123:6836/3425782983 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 --2- 172.21.15.64:0/1600120286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558e4afb3510 0x558e4afdd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 >> 172.21.15.64:0/1600120286 conn(0x558e4afafbc0 msgr2=0x558e4afbf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 shutdown_connections 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 1 -- 172.21.15.64:0/1600120286 wait complete. 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.134+0000 7faa1ff00740 20 client.15447 trim_cache size 0 max 16384 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:33:33.144 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24452, 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 121.39603342700001, 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 25, 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 94, 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 93, 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 183.21734683899999 2024-08-19T01:33:33.145 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faa000a2e10 msgr2=0x7faa000a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faa000a2e10 0x7faa000a52d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa080ed530 tx=0x7faa08047000 comp rx=0 tx=0).stop 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 msgr2=0x7faa181981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa181981a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faa08005ab0 tx=0x7faa080042f0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 shutdown_connections 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faa000a2e10 0x7faa000a52d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1810bea0 0x7faa1819ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa18103390 0x7faa181986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 --2- 172.21.15.64:0/999759741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa18102a70 0x7faa181981a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:33.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 >> 172.21.15.64:0/999759741 conn(0x7faa180fc4d0 msgr2=0x7faa180fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:33.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 shutdown_connections 2024-08-19T01:33:33.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:33.135+0000 7faa1db02640 1 -- 172.21.15.64:0/999759741 wait complete. 2024-08-19T01:33:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:32 smithi123 ceph-mon[21829]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-08-19T01:33:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:32 smithi161 ceph-mon[21757]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-08-19T01:33:33.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph tell mds.b status 2024-08-19T01:33:34.151 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 -- 172.21.15.64:0/2946158760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd481067b0 msgr2=0x7fdd48106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/2946158760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd481067b0 0x7fdd48106b90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009930 tx=0x7fdd4402a420 comp rx=0 tx=0).stop 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 -- 172.21.15.64:0/2946158760 shutdown_connections 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/2946158760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd48101150 0x7fdd4810d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/2946158760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd481007b0 0x7fdd48100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/2946158760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd481067b0 0x7fdd48106b90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 -- 172.21.15.64:0/2946158760 >> 172.21.15.64:0/2946158760 conn(0x7fdd480fc4d0 msgr2=0x7fdd480fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 -- 172.21.15.64:0/2946158760 shutdown_connections 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.638+0000 7fdd4fad5640 1 -- 172.21.15.64:0/2946158760 wait complete. 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 Processor -- start 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 -- start start 2024-08-19T01:33:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd481007b0 0x7fdd48198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd48101150 0x7fdd48198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd481067b0 0x7fdd4819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48110c50 con 0x7fdd48101150 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48110ad0 con 0x7fdd481067b0 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4fad5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48110dd0 con 0x7fdd481007b0 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.639+0000 7fdd4d070640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd48101150 0x7fdd48198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd48101150 0x7fdd48198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56072/0 (socket says 172.21.15.64:56072) 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 -- 172.21.15.64:0/3385097653 learned_addr learned my addr 172.21.15.64:0/3385097653 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d871640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd481007b0 0x7fdd48198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4e072640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd481067b0 0x7fdd4819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 -- 172.21.15.64:0/3385097653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd481007b0 msgr2=0x7fdd48198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd481007b0 0x7fdd48198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 -- 172.21.15.64:0/3385097653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd481067b0 msgr2=0x7fdd4819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd481067b0 0x7fdd4819cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 -- 172.21.15.64:0/3385097653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4819d2c0 con 0x7fdd48101150 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd48101150 0x7fdd48198850 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800b4a0 tx=0x7fdd3800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd3effd640 1 -- 172.21.15.64:0/3385097653 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38014070 con 0x7fdd48101150 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4819d5b0 con 0x7fdd48101150 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd3effd640 1 -- 172.21.15.64:0/3385097653 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd38010af0 con 0x7fdd48101150 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd3effd640 1 -- 172.21.15.64:0/3385097653 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38017d10 con 0x7fdd48101150 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.640+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd481a4e90 con 0x7fdd48101150 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd3effd640 1 -- 172.21.15.64:0/3385097653 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fdd38017070 con 0x7fdd48101150 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 Processor -- start 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 -- start start 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2630 0x55fb7ded2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7dedd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7dedfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fb7de37710 con 0x55fb7ded2f50 2024-08-19T01:33:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fb7ddcf770 con 0x55fb7ded2630 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.643+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb7de336d0 con 0x55fb7dedd810 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7dedd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd3effd640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdd340a2e10 0x7fdd340a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7dedd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56084/0 (socket says 172.21.15.64:56084) 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 -- 172.21.15.64:0/1905082445 learned_addr learned my addr 172.21.15.64:0/1905082445 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4e072640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdd340a2e10 0x7fdd340a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d070640 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7dedfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4e072640 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2630 0x55fb7ded2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 -- 172.21.15.64:0/1905082445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 msgr2=0x55fb7dedfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7dedfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 -- 172.21.15.64:0/1905082445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2630 msgr2=0x55fb7ded2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2630 0x55fb7ded2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 -- 172.21.15.64:0/1905082445 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb7ddd63e0 con 0x55fb7ded2f50 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd3effd640 1 -- 172.21.15.64:0/3385097653 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdd380ced10 con 0x7fdd48101150 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4d871640 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7dedd2d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fdd4402a940 tx=0x7fdd440316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1a036eef4936c0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd4e072640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdd340a2e10 0x7fdd340a52d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdd40014590 tx=0x7fdd40006090 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd33fff640 1 -- 172.21.15.64:0/1905082445 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44040070 con 0x55fb7ded2f50 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd33fff640 1 -- 172.21.15.64:0/1905082445 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd44005020 con 0x55fb7ded2f50 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.644+0000 7fdd33fff640 1 -- 172.21.15.64:0/1905082445 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44035040 con 0x55fb7ded2f50 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 -- 172.21.15.64:0/1905082445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 msgr2=0x55fb7dedd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7dedd2d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fdd4402a940 tx=0x7fdd440316e0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 -- 172.21.15.64:0/1905082445 shutdown_connections 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7dedfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7dedd2d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/1905082445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2630 0x55fb7ded2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 -- 172.21.15.64:0/1905082445 >> 172.21.15.64:0/1905082445 conn(0x55fb7decebc0 msgr2=0x55fb7decefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 -- 172.21.15.64:0/1905082445 shutdown_connections 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.645+0000 7fdd51ed3740 1 -- 172.21.15.64:0/1905082445 wait complete. 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 -- start start 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 Processor -- start 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7deda670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2f50 0x55fb7dedabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7dedb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fb7dee03e0 con 0x55fb7ded2630 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fb7ddf2970 con 0x55fb7dedd810 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb7dee0560 con 0x55fb7ded2f50 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd51ed3740 10 client.0 _ll_register_callbacks cb 0x55fb7db467b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd4d871640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7deda670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd4d871640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7deda670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56092/0 (socket says 172.21.15.64:56092) 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd4d871640 1 -- 172.21.15.64:0/704904723 learned_addr learned my addr 172.21.15.64:0/704904723 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd4d070640 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2f50 0x55fb7dedabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.646+0000 7fdd4e072640 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7dedb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd4d871640 1 -- 172.21.15.64:0/704904723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2f50 msgr2=0x55fb7dedabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd4d871640 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2f50 0x55fb7dedabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd4d871640 1 -- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 msgr2=0x55fb7dedb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd4d871640 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7dedb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd4d871640 1 -- 172.21.15.64:0/704904723 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb7dba92b0 con 0x55fb7ded2630 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd4d871640 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7deda670 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdd44031ba0 tx=0x7fdd440314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca3093834d8f2dc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd317fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd317fa640 1 -- 172.21.15.64:0/704904723 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4404d060 con 0x55fb7ded2630 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd317fa640 1 -- 172.21.15.64:0/704904723 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdd44004250 con 0x55fb7ded2630 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd44005020 con 0x55fb7ded2630 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd317fa640 1 -- 172.21.15.64:0/704904723 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdd44054050 con 0x55fb7ded2630 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd51ed3740 10 client.15459 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fb7deebf70 con 0x55fb7ded2630 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd317fa640 1 -- 172.21.15.64:0/704904723 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd44005020 con 0x55fb7ded2630 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.647+0000 7fdd317fa640 1 -- 172.21.15.64:0/704904723 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fdd440039c0 con 0x55fb7ded2630 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd51ed3740 10 client.15459 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd51ed3740 10 client.15459 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd51ed3740 20 client.15459 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7deee0c0 0x55fb7deef530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd51ed3740 4 client.15459 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb7de44c80 con 0x55fb7deee0c0 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd4d070640 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7deee0c0 0x55fb7deef530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd317fa640 10 client.15459 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.648+0000 7fdd4d070640 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7deee0c0 0x55fb7deef530 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6236b56062d7500 server_cookie=7a7e8699a7757c69 in_seq=0 out_seq=0 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd317fa640 1 -- 172.21.15.64:0/704904723 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fb7de44c80 con 0x55fb7deee0c0 2024-08-19T01:33:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd317fa640 10 client.15459 handle_command_reply: tid=0 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 client.15459 shutdown 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 msgr2=0x55fb7deda670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7deda670 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdd44031ba0 tx=0x7fdd440314b0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 shutdown_connections 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7deee0c0 0x55fb7deef530 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7dedb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2f50 0x55fb7dedabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/704904723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7deda670 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 >> 172.21.15.64:0/704904723 conn(0x55fb7decebc0 msgr2=0x55fb7ded48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd317fa640 0 client.15459 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 shutdown_connections 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 1 -- 172.21.15.64:0/704904723 wait complete. 2024-08-19T01:33:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.649+0000 7fdd51ed3740 20 client.15459 trim_cache size 0 max 16384 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 Processor -- start 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 -- start start 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7dedcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2f50 0x55fb7decdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7ded78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fb7deed870 con 0x55fb7ded2630 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fb7deedf40 con 0x55fb7ded2f50 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb7deee610 con 0x55fb7dedd810 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.667+0000 7fdd4d871640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7dedcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7dedcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56094/0 (socket says 172.21.15.64:56094) 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 -- 172.21.15.64:0/3119776334 learned_addr learned my addr 172.21.15.64:0/3119776334 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4e072640 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7ded78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 -- 172.21.15.64:0/3119776334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 msgr2=0x55fb7ded78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7ded78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 -- 172.21.15.64:0/3119776334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2f50 msgr2=0x55fb7decdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2f50 0x55fb7decdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2f50 0x55fb7decdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 -- 172.21.15.64:0/3119776334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb7dece9f0 con 0x55fb7ded2630 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4e072640 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7ded78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2f50 0x55fb7decdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd4d871640 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7dedcdd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdd44005de0 tx=0x7fdd44005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed4f12000f2cd2e server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd337fe640 1 -- 172.21.15.64:0/3119776334 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4405d070 con 0x55fb7ded2630 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd337fe640 1 -- 172.21.15.64:0/3119776334 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd44003ad0 con 0x55fb7ded2630 2024-08-19T01:33:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.668+0000 7fdd337fe640 1 -- 172.21.15.64:0/3119776334 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4404d070 con 0x55fb7ded2630 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3119776334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 msgr2=0x55fb7dedcdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7dedcdd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdd44005de0 tx=0x7fdd44005fd0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3119776334 shutdown_connections 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7dedd810 0x55fb7ded78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7ded2f50 0x55fb7decdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3119776334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2630 0x55fb7dedcdd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3119776334 >> 172.21.15.64:0/3119776334 conn(0x55fb7decebc0 msgr2=0x55fb7decd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3119776334 shutdown_connections 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3119776334 wait complete. 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 -- start start 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.669+0000 7fdd51ed3740 1 Processor -- start 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2630 0x55fb7def3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7def3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd51ed3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7def3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fb7deef3b0 con 0x55fb7ded2f50 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fb7deeece0 con 0x55fb7dedd810 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd51ed3740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb7deefa80 con 0x55fb7ded2630 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd51ed3740 10 client.0 _ll_register_callbacks cb 0x7fdd440315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7def3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7def3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56108/0 (socket says 172.21.15.64:56108) 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 -- 172.21.15.64:0/3638760220 learned_addr learned my addr 172.21.15.64:0/3638760220 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d871640 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2630 0x55fb7def3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4e072640 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7def3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 -- 172.21.15.64:0/3638760220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2630 msgr2=0x55fb7def3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2630 0x55fb7def3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 -- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 msgr2=0x55fb7def3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7def3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 -- 172.21.15.64:0/3638760220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb7deca130 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd4d070640 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7def3a50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdd3809aa40 tx=0x7fdd3806aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b0af7ea6e22284 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.670+0000 7fdd317fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd317fa640 1 -- 172.21.15.64:0/3638760220 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3801e040 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd317fa640 1 -- 172.21.15.64:0/3638760220 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdd38010ca0 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd44003ad0 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd317fa640 1 -- 172.21.15.64:0/3638760220 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdd3809f0f0 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 10 client.15465 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fb7df58880 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd317fa640 1 -- 172.21.15.64:0/3638760220 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd44003ad0 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd317fa640 1 -- 172.21.15.64:0/3638760220 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fdd38061030 con 0x55fb7ded2f50 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 10 client.15465 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 10 client.15465 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 20 client.15465 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7df5d480 0x55fb7df5c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 4 client.15465 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "status"}) -- 0x55fb7de3f190 con 0x55fb7df5d480 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.671+0000 7fdd4d871640 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7df5d480 0x55fb7df5c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd4d871640 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7df5d480 0x55fb7df5c190 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70c7481b9382a660 server_cookie=9e1f6be7c94f56cd in_seq=0 out_seq=0 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd317fa640 10 client.15465 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd317fa640 1 -- 172.21.15.64:0/3638760220 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fb7de3f190 con 0x55fb7df5d480 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd317fa640 10 client.15465 handle_command_reply: tid=0 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 client.15465 shutdown 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 msgr2=0x55fb7def3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7def3a50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdd3809aa40 tx=0x7fdd3806aba0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 shutdown_connections 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fb7df5d480 0x55fb7df5c190 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fb7dedd810 0x55fb7def3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fb7ded2f50 0x55fb7def3a50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd317fa640 0 client.15465 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 --2- 172.21.15.64:0/3638760220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb7ded2630 0x55fb7def3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 >> 172.21.15.64:0/3638760220 conn(0x55fb7decebc0 msgr2=0x55fb7decdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 shutdown_connections 2024-08-19T01:33:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 1 -- 172.21.15.64:0/3638760220 wait complete. 2024-08-19T01:33:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.672+0000 7fdd51ed3740 20 client.15465 trim_cache size 0 max 16384 2024-08-19T01:33:34.686 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "88c1a048-5dc9-11ef-bccf-c7b262605968", 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24476, 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 162.12631444900001, 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 25, 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 94, 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 93, 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 173.133376322 2024-08-19T01:33:34.687 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:33:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdd340a2e10 msgr2=0x7fdd340a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdd340a2e10 0x7fdd340a52d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdd40014590 tx=0x7fdd40006090 comp rx=0 tx=0).stop 2024-08-19T01:33:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd48101150 msgr2=0x7fdd48198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd48101150 0x7fdd48198850 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800b4a0 tx=0x7fdd3800b960 comp rx=0 tx=0).stop 2024-08-19T01:33:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 shutdown_connections 2024-08-19T01:33:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fdd340a2e10 0x7fdd340a52d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd481067b0 0x7fdd4819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd48101150 0x7fdd48198850 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 --2- 172.21.15.64:0/3385097653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd481007b0 0x7fdd48198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 >> 172.21.15.64:0/3385097653 conn(0x7fdd480fc4d0 msgr2=0x7fdd4806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 shutdown_connections 2024-08-19T01:33:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:34.673+0000 7fdd4fad5640 1 -- 172.21.15.64:0/3385097653 wait complete. 2024-08-19T01:33:35.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:34 smithi123 ceph-mon[21829]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-08-19T01:33:35.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:34 smithi161 ceph-mon[21757]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-08-19T01:33:35.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:34 smithi064 ceph-mon[18515]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-08-19T01:33:35.415 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-08-19T01:33:37.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:36 smithi123 ceph-mon[21829]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-08-19T01:33:37.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:36 smithi161 ceph-mon[21757]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-08-19T01:33:37.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:36 smithi064 ceph-mon[18515]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-08-19T01:33:39.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:39 smithi123 ceph-mon[21829]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-08-19T01:33:39.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:39 smithi161 ceph-mon[21757]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-08-19T01:33:39.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:39 smithi064 ceph-mon[18515]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-08-19T01:33:41.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:41 smithi123 ceph-mon[21829]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-08-19T01:33:41.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:41 smithi161 ceph-mon[21757]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-08-19T01:33:41.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:41 smithi064 ceph-mon[18515]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-08-19T01:33:43.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:43 smithi123 ceph-mon[21829]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 7 op/s 2024-08-19T01:33:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:43 smithi161 ceph-mon[21757]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 7 op/s 2024-08-19T01:33:43.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:43 smithi064 ceph-mon[18515]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 7 op/s 2024-08-19T01:33:45.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:45 smithi123 ceph-mon[21829]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:45.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:45 smithi161 ceph-mon[21757]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:45.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:45 smithi064 ceph-mon[18515]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:47.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:47 smithi123 ceph-mon[21829]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:33:47.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:47 smithi161 ceph-mon[21757]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:33:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:47 smithi064 ceph-mon[18515]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:33:49.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:49 smithi123 ceph-mon[21829]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:49.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:49 smithi161 ceph-mon[21757]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:49.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:49 smithi064 ceph-mon[18515]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:50.416 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 34.7 2024-08-19T01:33:50.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:33:50.681 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:51.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.643+0000 7f1f278e9640 1 -- 172.21.15.64:0/1266838809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f201022a0 msgr2=0x7f1f201026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.643+0000 7f1f278e9640 1 --2- 172.21.15.64:0/1266838809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f201022a0 0x7f1f201026e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009900 tx=0x7f1f1002a480 comp rx=0 tx=0).stop 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 -- 172.21.15.64:0/1266838809 shutdown_connections 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 --2- 172.21.15.64:0/1266838809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f20102c20 0x7f1f2010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 --2- 172.21.15.64:0/1266838809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f201022a0 0x7f1f201026e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 --2- 172.21.15.64:0/1266838809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f20100b50 0x7f1f20100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 -- 172.21.15.64:0/1266838809 >> 172.21.15.64:0/1266838809 conn(0x7f1f200fc500 msgr2=0x7f1f200fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 -- 172.21.15.64:0/1266838809 shutdown_connections 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 -- 172.21.15.64:0/1266838809 wait complete. 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 Processor -- start 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.644+0000 7f1f278e9640 1 -- start start 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f278e9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f20100b50 0x7f1f200fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f278e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f201022a0 0x7f1f200fcef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f278e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f20102c20 0x7f1f200fd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f278e9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1f20110c70 con 0x7f1f20100b50 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f278e9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f20110af0 con 0x7f1f20102c20 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f278e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1f20110df0 con 0x7f1f201022a0 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f20100b50 0x7f1f200fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f20100b50 0x7f1f200fc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55716/0 (socket says 172.21.15.64:55716) 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 -- 172.21.15.64:0/4053512692 learned_addr learned my addr 172.21.15.64:0/4053512692 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25e86640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f20102c20 0x7f1f200fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f24e84640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f201022a0 0x7f1f200fcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 -- 172.21.15.64:0/4053512692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f201022a0 msgr2=0x7f1f200fcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f201022a0 0x7f1f200fcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 -- 172.21.15.64:0/4053512692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f20102c20 msgr2=0x7f1f200fd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f20102c20 0x7f1f200fd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 -- 172.21.15.64:0/4053512692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f201a66a0 con 0x7f1f20100b50 2024-08-19T01:33:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.645+0000 7f1f25685640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f20100b50 0x7f1f200fc9b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00c920 tx=0x7f1f1c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.646+0000 7f1f167fc640 1 -- 172.21.15.64:0/4053512692 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c007c70 con 0x7f1f20100b50 2024-08-19T01:33:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.646+0000 7f1f167fc640 1 -- 172.21.15.64:0/4053512692 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f1c007e10 con 0x7f1f20100b50 2024-08-19T01:33:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.646+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f201a6990 con 0x7f1f20100b50 2024-08-19T01:33:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.646+0000 7f1f167fc640 1 -- 172.21.15.64:0/4053512692 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c018890 con 0x7f1f20100b50 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.646+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f201a6ef0 con 0x7f1f20100b50 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.647+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f20100f30 con 0x7f1f20100b50 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.648+0000 7f1f167fc640 1 -- 172.21.15.64:0/4053512692 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f1f1c018a70 con 0x7f1f20100b50 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.649+0000 7f1f167fc640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1f0c0a2e90 0x7f1f0c0a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.649+0000 7f1f167fc640 1 -- 172.21.15.64:0/4053512692 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1f1c0cfac0 con 0x7f1f20100b50 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.649+0000 7f1f24e84640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1f0c0a2e90 0x7f1f0c0a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.649+0000 7f1f24e84640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1f0c0a2e90 0x7f1f0c0a5350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009230 tx=0x7f1f100058d0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:51.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.653+0000 7f1f167fc640 1 -- 172.21.15.64:0/4053512692 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1f1c014030 con 0x7f1f20100b50 2024-08-19T01:33:51.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.876+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f20109dd0 con 0x7f1f20100b50 2024-08-19T01:33:51.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:51 smithi064 ceph-mon[18515]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:33:51.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.877+0000 7f1f167fc640 1 -- 172.21.15.64:0/4053512692 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11921 (secure 0 0 0) 0x7f1f1c09c0e0 con 0x7f1f20100b50 2024-08-19T01:33:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:33:51.878 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":25,"btime":"2024-08-19T01:33:12:555328+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:33:12.505142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:33:51.879 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 25 2024-08-19T01:33:51.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1f0c0a2e90 msgr2=0x7f1f0c0a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:51.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1f0c0a2e90 0x7f1f0c0a5350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009230 tx=0x7f1f100058d0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f20100b50 msgr2=0x7f1f200fc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:51.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f20100b50 0x7f1f200fc9b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00c920 tx=0x7f1f1c00cde0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 shutdown_connections 2024-08-19T01:33:51.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1f0c0a2e90 0x7f1f0c0a5350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f20102c20 0x7f1f200fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f201022a0 0x7f1f200fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 --2- 172.21.15.64:0/4053512692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f20100b50 0x7f1f200fc9b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:51.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 >> 172.21.15.64:0/4053512692 conn(0x7f1f200fc500 msgr2=0x7f1f20069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:51.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 shutdown_connections 2024-08-19T01:33:51.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:51.879+0000 7f1f278e9640 1 -- 172.21.15.64:0/4053512692 wait complete. 2024-08-19T01:33:51.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:51 smithi123 ceph-mon[21829]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:33:51.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:51 smithi161 ceph-mon[21757]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-08-19T01:33:52.677 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 154.7 --await-for 45.0' 2024-08-19T01:33:52.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 154.7 --await-for 45.0 2024-08-19T01:33:52.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:52 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4053512692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:52.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:52 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4053512692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:52.954 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:33:52.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:52 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4053512692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 -- 172.21.15.64:0/2251546296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c107fd0 msgr2=0x7f8a3c10e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/2251546296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c107fd0 0x7f8a3c10e970 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009cd0 tx=0x7f8a3402a910 comp rx=0 tx=0).stop 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 -- 172.21.15.64:0/2251546296 shutdown_connections 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/2251546296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c107fd0 0x7f8a3c10e970 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/2251546296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a3c103410 0x7f8a3c1078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/2251546296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a3c102a60 0x7f8a3c102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 -- 172.21.15.64:0/2251546296 >> 172.21.15.64:0/2251546296 conn(0x7f8a3c0fc500 msgr2=0x7f8a3c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 -- 172.21.15.64:0/2251546296 shutdown_connections 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 -- 172.21.15.64:0/2251546296 wait complete. 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 Processor -- start 2024-08-19T01:33:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.461+0000 7f8a43e1d640 1 -- start start 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a43e1d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a3c102a60 0x7f8a3c1006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a43e1d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c103410 0x7f8a3c100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a43e1d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a3c107fd0 0x7f8a3c101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a43e1d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c111e60 con 0x7f8a3c103410 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a43e1d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c111ce0 con 0x7f8a3c107fd0 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a43e1d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c111fe0 con 0x7f8a3c102a60 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c103410 0x7f8a3c100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c103410 0x7f8a3c100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55746/0 (socket says 172.21.15.64:55746) 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a41bb9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a3c102a60 0x7f8a3c1006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 -- 172.21.15.64:0/293939837 learned_addr learned my addr 172.21.15.64:0/293939837 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a423ba640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a3c107fd0 0x7f8a3c101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 -- 172.21.15.64:0/293939837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a3c102a60 msgr2=0x7f8a3c1006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a3c102a60 0x7f8a3c1006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 -- 172.21.15.64:0/293939837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a3c107fd0 msgr2=0x7f8a3c101140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a3c107fd0 0x7f8a3c101140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 -- 172.21.15.64:0/293939837 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3c1a6750 con 0x7f8a3c103410 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a413b8640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c103410 0x7f8a3c100c00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c006fd0 tx=0x7f8a2c00b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.462+0000 7f8a32ffd640 1 -- 172.21.15.64:0/293939837 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c015020 con 0x7f8a3c103410 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.463+0000 7f8a32ffd640 1 -- 172.21.15.64:0/293939837 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a2c011d10 con 0x7f8a3c103410 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.463+0000 7f8a32ffd640 1 -- 172.21.15.64:0/293939837 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c00f2e0 con 0x7f8a3c103410 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.463+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3c1a6a40 con 0x7f8a3c103410 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.463+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3c1a6f80 con 0x7f8a3c103410 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a32ffd640 1 -- 172.21.15.64:0/293939837 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f8a2c018020 con 0x7f8a3c103410 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 Processor -- start 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 -- start start 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e72b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e735dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e738700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564f5e690a70 con 0x564f5e72ba50 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564f5e628780 con 0x564f5e72b130 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a4621b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f5e68ca30 con 0x564f5e736310 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.466+0000 7f8a41bb9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e735dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e735dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55748/0 (socket says 172.21.15.64:55748) 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 -- 172.21.15.64:0/2520675449 learned_addr learned my addr 172.21.15.64:0/2520675449 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a32ffd640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8a280a2e90 0x7f8a280a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a413b8640 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e738700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a423ba640 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e72b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 -- 172.21.15.64:0/2520675449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 msgr2=0x564f5e738700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a423ba640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8a280a2e90 0x7f8a280a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e738700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 -- 172.21.15.64:0/2520675449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 msgr2=0x564f5e72b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e72b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 -- 172.21.15.64:0/2520675449 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5e402b90 con 0x564f5e72ba50 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a32ffd640 1 -- 172.21.15.64:0/293939837 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8a2c0cf1c0 con 0x7f8a3c103410 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e735dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800ad00 tx=0x7f8a3800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8ec2ff8d1081bf server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a27fff640 1 -- 172.21.15.64:0/2520675449 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3801d070 con 0x564f5e72ba50 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a423ba640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8a280a2e90 0x7f8a280a5350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a34006010 tx=0x7f8a34009800 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a27fff640 1 -- 172.21.15.64:0/2520675449 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a38018b20 con 0x564f5e72ba50 2024-08-19T01:33:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.467+0000 7f8a27fff640 1 -- 172.21.15.64:0/2520675449 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38010bc0 con 0x564f5e72ba50 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 -- 172.21.15.64:0/2520675449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 msgr2=0x564f5e735dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e735dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800ad00 tx=0x7f8a3800f790 comp rx=0 tx=0).stop 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 -- 172.21.15.64:0/2520675449 shutdown_connections 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e738700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e735dd0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 --2- 172.21.15.64:0/2520675449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e72b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 -- 172.21.15.64:0/2520675449 >> 172.21.15.64:0/2520675449 conn(0x564f5e7276c0 msgr2=0x564f5e727aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 -- 172.21.15.64:0/2520675449 shutdown_connections 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.468+0000 7f8a4621b740 1 -- 172.21.15.64:0/2520675449 wait complete. 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 -- start start 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 Processor -- start 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e733130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e733670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e733bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564f5e738ee0 con 0x564f5e72ba50 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564f5e64b990 con 0x564f5e72b130 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.469+0000 7f8a4621b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f5e739060 con 0x564f5e736310 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a4621b740 10 client.0 _ll_register_callbacks cb 0x564f5e3a0090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e733670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e733670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55764/0 (socket says 172.21.15.64:55764) 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 -- 172.21.15.64:0/3272634265 learned_addr learned my addr 172.21.15.64:0/3272634265 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a423ba640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e733bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e733130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 -- 172.21.15.64:0/3272634265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 msgr2=0x564f5e733bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e733bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 -- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 msgr2=0x564f5e733130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e733130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 -- 172.21.15.64:0/3272634265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5e64a6c0 con 0x564f5e72ba50 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a423ba640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e733bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e733130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a413b8640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e733670 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c097650 tx=0x7f8a2c09afc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1130af7ce55581 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a257fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a257fa640 1 -- 172.21.15.64:0/3272634265 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c010ec0 con 0x564f5e72ba50 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a257fa640 1 -- 172.21.15.64:0/3272634265 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8a2c097860 con 0x564f5e72ba50 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.470+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a38018b20 con 0x564f5e72ba50 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a257fa640 1 -- 172.21.15.64:0/3272634265 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8a2c09b8f0 con 0x564f5e72ba50 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 10 client.15483 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564f5e744ac0 con 0x564f5e72ba50 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a257fa640 1 -- 172.21.15.64:0/3272634265 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a38018b20 con 0x564f5e72ba50 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a257fa640 1 -- 172.21.15.64:0/3272634265 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f8a2c09f180 con 0x564f5e72ba50 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 10 client.15483 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 10 client.15483 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 20 client.15483 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e745e10 0x564f5e748240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 4 client.15483 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f5e69e1a0 con 0x564f5e745e10 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.471+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e745e10 0x564f5e748240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.472+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e745e10 0x564f5e748240 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d016bd70c986fb9a server_cookie=8c6bc2213b7f5c1e in_seq=0 out_seq=0 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.472+0000 7f8a257fa640 10 client.15483 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a257fa640 1 -- 172.21.15.64:0/3272634265 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f5e69e1a0 con 0x564f5e745e10 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a257fa640 10 client.15483 handle_command_reply: tid=0 2024-08-19T01:33:53.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 client.15483 shutdown 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 msgr2=0x564f5e733670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e733670 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c097650 tx=0x7f8a2c09afc0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 shutdown_connections 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e733bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72ba50 0x564f5e733670 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e745e10 0x564f5e748240 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3272634265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72b130 0x564f5e733130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a257fa640 0 client.15483 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 >> 172.21.15.64:0/3272634265 conn(0x564f5e7276c0 msgr2=0x564f5e72d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 shutdown_connections 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 1 -- 172.21.15.64:0/3272634265 wait complete. 2024-08-19T01:33:53.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.473+0000 7f8a4621b740 20 client.15483 trim_cache size 0 max 16384 2024-08-19T01:33:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.493+0000 7f8a4621b740 1 Processor -- start 2024-08-19T01:33:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.493+0000 7f8a4621b740 1 -- start start 2024-08-19T01:33:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e72b130 0x564f5e7320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e72dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e736310 0x564f5e74dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a4621b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1b3850 con 0x564f5e736310 2024-08-19T01:33:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a4621b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1b1120 con 0x564f5e72ba50 2024-08-19T01:33:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a4621b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1aef70 con 0x564f5e72b130 2024-08-19T01:33:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a423ba640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e736310 0x564f5e74dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a423ba640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e736310 0x564f5e74dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55770/0 (socket says 172.21.15.64:55770) 2024-08-19T01:33:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a423ba640 1 -- 172.21.15.64:0/1403071850 learned_addr learned my addr 172.21.15.64:0/1403071850 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a413b8640 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e72dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e72b130 0x564f5e7320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a413b8640 1 -- 172.21.15.64:0/1403071850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e72b130 msgr2=0x564f5e7320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a413b8640 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e72b130 0x564f5e7320d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a413b8640 1 -- 172.21.15.64:0/1403071850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e736310 msgr2=0x564f5e74dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a413b8640 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e736310 0x564f5e74dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a413b8640 1 -- 172.21.15.64:0/1403071850 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5e735c80 con 0x564f5e72ba50 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.494+0000 7f8a413b8640 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e72dbe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0994a0 tx=0x7f8a2c0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c26908f1f2affe server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.495+0000 7f8a277fe640 1 -- 172.21.15.64:0/1403071850 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c09ad30 con 0x564f5e72ba50 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.495+0000 7f8a277fe640 1 -- 172.21.15.64:0/1403071850 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a2c010ec0 con 0x564f5e72ba50 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.495+0000 7f8a277fe640 1 -- 172.21.15.64:0/1403071850 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c09f180 con 0x564f5e72ba50 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.495+0000 7f8a4621b740 1 -- 172.21.15.64:0/1403071850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 msgr2=0x564f5e72dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.495+0000 7f8a4621b740 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e72dbe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0994a0 tx=0x7f8a2c0108e0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 -- 172.21.15.64:0/1403071850 shutdown_connections 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e736310 0x564f5e74dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e72dbe0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 --2- 172.21.15.64:0/1403071850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e72b130 0x564f5e7320d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 -- 172.21.15.64:0/1403071850 >> 172.21.15.64:0/1403071850 conn(0x564f5e7276c0 msgr2=0x564f5e7351b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 -- 172.21.15.64:0/1403071850 shutdown_connections 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 -- 172.21.15.64:0/1403071850 wait complete. 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 -- start start 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 Processor -- start 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.496+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72b130 0x564f5e73e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e73e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a4621b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e73b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a4621b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1af5e0 con 0x564f5e72b130 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a4621b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1b0a50 con 0x564f5e72ba50 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a4621b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1afcb0 con 0x564f5e736310 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a4621b740 10 client.0 _ll_register_callbacks cb 0x7f8a2c0107e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72b130 0x564f5e73e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72b130 0x564f5e73e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55786/0 (socket says 172.21.15.64:55786) 2024-08-19T01:33:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 -- 172.21.15.64:0/3754659118 learned_addr learned my addr 172.21.15.64:0/3754659118 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a423ba640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e73b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a413b8640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e73e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 -- 172.21.15.64:0/3754659118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 msgr2=0x564f5e73b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e73b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 -- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 msgr2=0x564f5e73e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e73e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 -- 172.21.15.64:0/3754659118 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5e73b7c0 con 0x564f5e72b130 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a413b8640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e73e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a41bb9640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72b130 0x564f5e73e2b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800fc90 tx=0x7f8a3800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbeffe8aa2a489be server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a257fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a257fa640 1 -- 172.21.15.64:0/3754659118 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3801d040 con 0x564f5e72b130 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a257fa640 1 -- 172.21.15.64:0/3754659118 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8a38018660 con 0x564f5e72b130 2024-08-19T01:33:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.497+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a2c010ec0 con 0x564f5e72b130 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a257fa640 1 -- 172.21.15.64:0/3754659118 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8a3803a050 con 0x564f5e72b130 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 10 client.15489 fetch_fsmap learned FSMap version 25 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564f5e7b1120 con 0x564f5e72b130 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a257fa640 1 -- 172.21.15.64:0/3754659118 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a2c010ec0 con 0x564f5e72b130 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a257fa640 1 -- 172.21.15.64:0/3754659118 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f8a38010730 con 0x564f5e72b130 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 10 client.15489 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 10 client.15489 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 20 client.15489 populate_metadata read hostname 'smithi064' 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e7b1630 0x564f5e7b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 4 client.15489 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.7, "await_for": 45.0}] 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 154.7, "await_for": 45.0}) -- 0x564f5e6984f0 con 0x564f5e7b1630 2024-08-19T01:33:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.498+0000 7f8a413b8640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e7b1630 0x564f5e7b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:33:53.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.499+0000 7f8a413b8640 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e7b1630 0x564f5e7b3a20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ded88b5139f79da server_cookie=ff9e83a80136273 in_seq=0 out_seq=0 2024-08-19T01:33:53.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.499+0000 7f8a257fa640 10 client.15489 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a257fa640 1 -- 172.21.15.64:0/3754659118 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x564f5e6984f0 con 0x564f5e7b1630 2024-08-19T01:33:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a257fa640 10 client.15489 handle_command_reply: tid=0 2024-08-19T01:33:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 client.15489 shutdown 2024-08-19T01:33:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72b130 msgr2=0x564f5e73e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72b130 0x564f5e73e2b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800fc90 tx=0x7f8a3800fd40 comp rx=0 tx=0).stop 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 shutdown_connections 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f5e736310 0x564f5e73b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x564f5e7b1630 0x564f5e7b3a20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f5e72ba50 0x564f5e73e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 --2- 172.21.15.64:0/3754659118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564f5e72b130 0x564f5e73e2b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 >> 172.21.15.64:0/3754659118 conn(0x564f5e7276c0 msgr2=0x564f5e734990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a257fa640 0 client.15489 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 shutdown_connections 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 1 -- 172.21.15.64:0/3754659118 wait complete. 2024-08-19T01:33:53.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.511+0000 7f8a4621b740 20 client.15489 trim_cache size 0 max 16384 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 25, 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 2, 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "9fa8f803": { 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 2, 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:33:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 154.7, 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:33:53.515 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:33:53.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8a280a2e90 msgr2=0x7f8a280a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8a280a2e90 0x7f8a280a5350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a34006010 tx=0x7f8a34009800 comp rx=0 tx=0).stop 2024-08-19T01:33:53.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c103410 msgr2=0x7f8a3c100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:33:53.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c103410 0x7f8a3c100c00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c006fd0 tx=0x7f8a2c00b820 comp rx=0 tx=0).stop 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 shutdown_connections 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8a280a2e90 0x7f8a280a5350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a3c107fd0 0x7f8a3c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a3c103410 0x7f8a3c100c00 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 --2- 172.21.15.64:0/293939837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a3c102a60 0x7f8a3c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 >> 172.21.15.64:0/293939837 conn(0x7f8a3c0fc500 msgr2=0x7f8a3c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 shutdown_connections 2024-08-19T01:33:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:33:53.512+0000 7f8a43e1d640 1 -- 172.21.15.64:0/293939837 wait complete. 2024-08-19T01:33:53.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:53 smithi123 ceph-mon[21829]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:53.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:53 smithi161 ceph-mon[21757]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:53.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:53 smithi064 ceph-mon[18515]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-08-19T01:33:54.403 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9fa8f803', quiesce took 4.0 seconds. Will release after: 30.700000000000003 2024-08-19T01:33:55.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:55 smithi123 ceph-mon[21829]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:55.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:55 smithi161 ceph-mon[21757]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:55.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:55 smithi064 ceph-mon[18515]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:33:57.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:57 smithi123 ceph-mon[21829]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-08-19T01:33:57.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:57 smithi161 ceph-mon[21757]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-08-19T01:33:57.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:57 smithi064 ceph-mon[18515]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-08-19T01:33:59.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:33:59 smithi123 ceph-mon[21829]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-08-19T01:33:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:33:59 smithi161 ceph-mon[21757]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-08-19T01:33:59.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:33:59 smithi064 ceph-mon[18515]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-08-19T01:34:01.234 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:34:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:34:00.765+0000 7fd4c0a93640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:34:01.234 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:34:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:34:00.765+0000 7fd4c0a93640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:34:01.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:34:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:34:00.765+0000 7fd4c0a93640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:34:01.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:34:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:34:00.765+0000 7fd4c0a93640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:34:01.235 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:34:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x[18742]: 2024-08-19T01:34:00.765+0000 7fd4c0a93640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-19T01:34:01.902 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:01 smithi064 ceph-mon[18515]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-08-19T01:34:01.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:01 smithi123 ceph-mon[21829]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-08-19T01:34:01.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:01 smithi161 ceph-mon[21757]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-08-19T01:34:03.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:03 smithi123 ceph-mon[21829]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-08-19T01:34:03.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:03 smithi123 ceph-mon[21829]: mgrmap e25: x(active, since 48s), standbys: z, y 2024-08-19T01:34:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:03 smithi161 ceph-mon[21757]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-08-19T01:34:03.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:03 smithi161 ceph-mon[21757]: mgrmap e25: x(active, since 48s), standbys: z, y 2024-08-19T01:34:03.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:03 smithi064 ceph-mon[18515]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-08-19T01:34:03.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:03 smithi064 ceph-mon[18515]: mgrmap e25: x(active, since 48s), standbys: z, y 2024-08-19T01:34:05.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:05 smithi123 ceph-mon[21829]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-08-19T01:34:05.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:05 smithi161 ceph-mon[21757]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-08-19T01:34:05.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:05 smithi064 ceph-mon[18515]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-08-19T01:34:07.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:07 smithi123 ceph-mon[21829]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:07.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:07 smithi161 ceph-mon[21757]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:07.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:07 smithi064 ceph-mon[18515]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:09.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:09 smithi123 ceph-mon[21829]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-08-19T01:34:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:09 smithi161 ceph-mon[21757]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-08-19T01:34:09.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:09 smithi064 ceph-mon[18515]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-08-19T01:34:11.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:11 smithi123 ceph-mon[21829]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:11 smithi161 ceph-mon[21757]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:12.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:11 smithi064 ceph-mon[18515]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:13.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:13 smithi161 ceph-mon[21757]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-08-19T01:34:13.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:13 smithi064 ceph-mon[18515]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-08-19T01:34:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:13 smithi123 ceph-mon[21829]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-08-19T01:34:15.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:15 smithi161 ceph-mon[21757]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-08-19T01:34:15.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:15 smithi064 ceph-mon[18515]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-08-19T01:34:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:15 smithi123 ceph-mon[21829]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-08-19T01:34:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:17 smithi161 ceph-mon[21757]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:17.967 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:17 smithi123 ceph-mon[21829]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:17.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:17 smithi064 ceph-mon[18515]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-08-19T01:34:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:18 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:34:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:18 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:34:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:18 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:34:18.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:18 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:34:18.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:18 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:34:18.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:18 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:34:18.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:18 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:34:18.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:18 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:34:19.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:18 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:34:19.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:18 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:34:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:18 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:34:19.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:18 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:34:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:19 smithi161 ceph-mon[21757]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-08-19T01:34:19.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:19 smithi064 ceph-mon[18515]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-08-19T01:34:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:19 smithi123 ceph-mon[21829]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-08-19T01:34:21.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:21 smithi161 ceph-mon[21757]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-08-19T01:34:21.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:21 smithi064 ceph-mon[18515]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-08-19T01:34:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:21 smithi123 ceph-mon[21829]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-08-19T01:34:23.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:23 smithi161 ceph-mon[21757]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-08-19T01:34:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:23 smithi064 ceph-mon[18515]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-08-19T01:34:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:23 smithi123 ceph-mon[21829]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-08-19T01:34:25.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:34:25.368 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:34:25.644 INFO:tasks.workunit.client.0.smithi064.stderr:Updating files: 93% (12041/12849) Updating files: 94% (12079/12849) Updating files: 95% (12207/12849) Updating files: 96% (12336/12849) Updating files: 97% (12464/12849) Updating files: 98% (12593/12849) Updating files: 99% (12721/12849) Updating files: 100% (12849/12849) Updating files: 100% (12849/12849), done. 2024-08-19T01:34:25.765 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:25 smithi064 ceph-mon[18515]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-08-19T01:34:25.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:25 smithi161 ceph-mon[21757]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-08-19T01:34:26.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 -- 172.21.15.64:0/2268627029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 msgr2=0x7f291c10d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 --2- 172.21.15.64:0/2268627029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c10d770 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2914009ca0 tx=0x7f291402a950 comp rx=0 tx=0).stop 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 -- 172.21.15.64:0/2268627029 shutdown_connections 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 --2- 172.21.15.64:0/2268627029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c10d770 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 --2- 172.21.15.64:0/2268627029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f291c1022a0 0x7f291c1026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 --2- 172.21.15.64:0/2268627029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f291c100b50 0x7f291c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 -- 172.21.15.64:0/2268627029 >> 172.21.15.64:0/2268627029 conn(0x7f291c0fc500 msgr2=0x7f291c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 -- 172.21.15.64:0/2268627029 shutdown_connections 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.004+0000 7f29225b9640 1 -- 172.21.15.64:0/2268627029 wait complete. 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 Processor -- start 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 -- start start 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f291c100b50 0x7f291c197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:26.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f291c1022a0 0x7f291c1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c19ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f291c110cd0 con 0x7f291c102c20 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f291c110b50 con 0x7f291c100b50 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f29225b9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f291c110e50 con 0x7f291c1022a0 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f2920b56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c19ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f2920b56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c19ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:58094/0 (socket says 172.21.15.64:58094) 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f2920b56640 1 -- 172.21.15.64:0/3504377147 learned_addr learned my addr 172.21.15.64:0/3504377147 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f291bfff640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f291c100b50 0x7f291c197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.005+0000 7f291b7fe640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f291c1022a0 0x7f291c1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f2920b56640 1 -- 172.21.15.64:0/3504377147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f291c1022a0 msgr2=0x7f291c1976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f2920b56640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f291c1022a0 0x7f291c1976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f2920b56640 1 -- 172.21.15.64:0/3504377147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f291c100b50 msgr2=0x7f291c197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f2920b56640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f291c100b50 0x7f291c197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f2920b56640 1 -- 172.21.15.64:0/3504377147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c19c120 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f2920b56640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c19ba40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2914006010 tx=0x7f291402f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f29197fa640 1 -- 172.21.15.64:0/3504377147 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914034040 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f291c19c3b0 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f29197fa640 1 -- 172.21.15.64:0/3504377147 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2914003e00 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f29197fa640 1 -- 172.21.15.64:0/3504377147 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914045480 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.006+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f291c069ca0 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.007+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f291c101ee0 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.008+0000 7f29197fa640 1 -- 172.21.15.64:0/3504377147 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f2914037020 con 0x7f291c102c20 2024-08-19T01:34:26.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.009+0000 7f29197fa640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29040a2ea0 0x7f29040a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:26.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.009+0000 7f29197fa640 1 -- 172.21.15.64:0/3504377147 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f29140f0a10 con 0x7f291c102c20 2024-08-19T01:34:26.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.009+0000 7f291bfff640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29040a2ea0 0x7f29040a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:26.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.009+0000 7f291bfff640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29040a2ea0 0x7f29040a5360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f290c009990 tx=0x7f290c009110 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:26.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.013+0000 7f29197fa640 1 -- 172.21.15.64:0/3504377147 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f29140bd150 con 0x7f291c102c20 2024-08-19T01:34:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:25 smithi123 ceph-mon[21829]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-08-19T01:34:26.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.225+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f291c1984d0 con 0x7f291c102c20 2024-08-19T01:34:26.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.226+0000 7f29197fa640 1 -- 172.21.15.64:0/3504377147 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11921 (secure 0 0 0) 0x7f29140bca50 con 0x7f291c102c20 2024-08-19T01:34:26.228 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:34:26.228 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":25,"btime":"2024-08-19T01:33:12:555328+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:33:12.505142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:34:26.228 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 25 2024-08-19T01:34:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29040a2ea0 msgr2=0x7f29040a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29040a2ea0 0x7f29040a5360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f290c009990 tx=0x7f290c009110 comp rx=0 tx=0).stop 2024-08-19T01:34:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 msgr2=0x7f291c19ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c19ba40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2914006010 tx=0x7f291402f6f0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 shutdown_connections 2024-08-19T01:34:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29040a2ea0 0x7f29040a5360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f291c102c20 0x7f291c19ba40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f291c1022a0 0x7f291c1976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 --2- 172.21.15.64:0/3504377147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f291c100b50 0x7f291c197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:26.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 >> 172.21.15.64:0/3504377147 conn(0x7f291c0fc500 msgr2=0x7f291c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:26.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 shutdown_connections 2024-08-19T01:34:26.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:26.229+0000 7f29225b9640 1 -- 172.21.15.64:0/3504377147 wait complete. 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr:Note: switching to '78215f5a1cf5eaf321d506a66539942e12851f34'. 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr:state without impacting any branches by switching back to a branch. 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-19T01:34:26.452 INFO:tasks.workunit.client.0.smithi064.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr: git switch -c 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr:Or undo this operation with: 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr: git switch - 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-08-19T01:34:26.453 INFO:tasks.workunit.client.0.smithi064.stderr:HEAD is now at 78215f5a1cf Merge pull request #59291 from zdover23/wip-doc-2024-08-19-backport-59290-to-squid 2024-08-19T01:34:26.458 DEBUG:teuthology.orchestra.run.smithi064:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-19T01:34:26.533 INFO:tasks.workunit.client.0.smithi064.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-19T01:34:26.535 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-19T01:34:26.535 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-19T01:34:26.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:26 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3504377147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:34:26.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:26 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3504377147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:34:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:26 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3504377147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:34:27.346 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 9fa8f803 --release --await' 2024-08-19T01:34:27.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 9fa8f803 --release --await 2024-08-19T01:34:27.608 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:34:27.862 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-19T01:34:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:27 smithi161 ceph-mon[21757]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:34:27.951 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-19T01:34:27.953 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:27 smithi064 ceph-mon[18515]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:34:27.992 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-19T01:34:27.994 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-19T01:34:27.994 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-19T01:34:28.037 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-19T01:34:28.040 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:34:28.041 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-19T01:34:28.056 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-08-19T01:34:28.057 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-08-19T01:34:28.057 DEBUG:teuthology.orchestra.run.smithi064:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-08-19T01:34:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:27 smithi123 ceph-mon[21829]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-08-19T01:34:28.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 -- 172.21.15.64:0/1019000648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5114102250 msgr2=0x7f511410e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 --2- 172.21.15.64:0/1019000648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5114102250 0x7f511410e780 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f510c009cd0 tx=0x7f510c02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 -- 172.21.15.64:0/1019000648 shutdown_connections 2024-08-19T01:34:28.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 --2- 172.21.15.64:0/1019000648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5114102250 0x7f511410e780 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 --2- 172.21.15.64:0/1019000648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51141018b0 0x7f5114101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 --2- 172.21.15.64:0/1019000648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51141078b0 0x7f5114107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 -- 172.21.15.64:0/1019000648 >> 172.21.15.64:0/1019000648 conn(0x7f51140fd5f0 msgr2=0x7f51140ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 -- 172.21.15.64:0/1019000648 shutdown_connections 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.183+0000 7f511aab0640 1 -- 172.21.15.64:0/1019000648 wait complete. 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 Processor -- start 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 -- start start 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51141018b0 0x7f51141982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5114102250 0x7f5114198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51141078b0 0x7f511419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5114111da0 con 0x7f51141078b0 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5114111c20 con 0x7f51141018b0 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511aab0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5114111f20 con 0x7f5114102250 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511904d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51141078b0 0x7f511419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511904d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51141078b0 0x7f511419cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:58112/0 (socket says 172.21.15.64:58112) 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.184+0000 7f511904d640 1 -- 172.21.15.64:0/775810995 learned_addr learned my addr 172.21.15.64:0/775810995 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:34:28.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511884c640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51141018b0 0x7f51141982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f510bfff640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5114102250 0x7f5114198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511904d640 1 -- 172.21.15.64:0/775810995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5114102250 msgr2=0x7f5114198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511904d640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5114102250 0x7f5114198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511904d640 1 -- 172.21.15.64:0/775810995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51141018b0 msgr2=0x7f51141982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511904d640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51141018b0 0x7f51141982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511904d640 1 -- 172.21.15.64:0/775810995 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f511419d290 con 0x7f51141078b0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f510bfff640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5114102250 0x7f5114198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511904d640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51141078b0 0x7f511419cbb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f510c009ca0 tx=0x7f510c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f5109ffb640 1 -- 172.21.15.64:0/775810995 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c004240 con 0x7f51141078b0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f511419d520 con 0x7f51141078b0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f5109ffb640 1 -- 172.21.15.64:0/775810995 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f510c0305a0 con 0x7f51141078b0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f5109ffb640 1 -- 172.21.15.64:0/775810995 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c03a660 con 0x7f51141078b0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.185+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51141a4e60 con 0x7f51141078b0 2024-08-19T01:34:28.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.187+0000 7f5109ffb640 1 -- 172.21.15.64:0/775810995 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f510c030750 con 0x7f51141078b0 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 Processor -- start 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 -- start start 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7be4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f5109ffb640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f50fc0a2ea0 0x7f50fc0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bf1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5583b7b4ac60 con 0x5583b7be52b0 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5583b7ae2b90 con 0x5583b7befb70 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511ceae740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5583b7b46c20 con 0x5583b7be4990 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511884c640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f50fc0a2ea0 0x7f50fc0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f5109ffb640 1 -- 172.21.15.64:0/775810995 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f510c02e030 con 0x7f51141078b0 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f510bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f510bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bef630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:58126/0 (socket says 172.21.15.64:58126) 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f510bfff640 1 -- 172.21.15.64:0/3554405069 learned_addr learned my addr 172.21.15.64:0/3554405069 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f510bfff640 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7be4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.188+0000 7f511904d640 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bf1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f510bfff640 1 -- 172.21.15.64:0/3554405069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 msgr2=0x5583b7be4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f510bfff640 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7be4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f510bfff640 1 -- 172.21.15.64:0/3554405069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 msgr2=0x5583b7bf1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f510bfff640 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bf1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f510bfff640 1 -- 172.21.15.64:0/3554405069 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b785a580 con 0x5583b7be52b0 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f511884c640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f50fc0a2ea0 0x7f50fc0a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5110009890 tx=0x7f5110008040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f511904d640 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bf1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f510bfff640 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bef630 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5104006fd0 tx=0x7f510400c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b374bd439c9b0547 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f50faffd640 1 -- 172.21.15.64:0/3554405069 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510401c070 con 0x5583b7be52b0 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f50faffd640 1 -- 172.21.15.64:0/3554405069 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5104017af0 con 0x5583b7be52b0 2024-08-19T01:34:28.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f50faffd640 1 -- 172.21.15.64:0/3554405069 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510400f040 con 0x5583b7be52b0 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f511ceae740 1 -- 172.21.15.64:0/3554405069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 msgr2=0x5583b7bef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.189+0000 7f511ceae740 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bef630 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5104006fd0 tx=0x7f510400c230 comp rx=0 tx=0).stop 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- 172.21.15.64:0/3554405069 shutdown_connections 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bf1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bef630 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 --2- 172.21.15.64:0/3554405069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7be4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- 172.21.15.64:0/3554405069 >> 172.21.15.64:0/3554405069 conn(0x5583b7be0f20 msgr2=0x5583b7be1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- 172.21.15.64:0/3554405069 shutdown_connections 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- 172.21.15.64:0/3554405069 wait complete. 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- start start 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 Processor -- start 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7becae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be52b0 0x5583b7bed020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5583b7bf2740 con 0x5583b7be4990 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5583b7ae87c0 con 0x5583b7befb70 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.190+0000 7f511ceae740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5583b7bf28c0 con 0x5583b7be52b0 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511ceae740 10 client.0 _ll_register_callbacks cb 0x5583b7af1f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7becae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7becae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:58128/0 (socket says 172.21.15.64:58128) 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 -- 172.21.15.64:0/228896728 learned_addr learned my addr 172.21.15.64:0/228896728 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511904d640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f510bfff640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be52b0 0x5583b7bed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 -- 172.21.15.64:0/228896728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be52b0 msgr2=0x5583b7bed020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be52b0 0x5583b7bed020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 -- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 msgr2=0x5583b7bed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 -- 172.21.15.64:0/228896728 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583b7924960 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511904d640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511884c640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7becae0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5110009670 tx=0x7f51100047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9929d967b05b78b7 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f50f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f50f3fff640 1 -- 172.21.15.64:0/228896728 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110024060 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f50f3fff640 1 -- 172.21.15.64:0/228896728 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f511001f400 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5104017af0 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f50f3fff640 1 -- 172.21.15.64:0/228896728 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f511002b050 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511ceae740 10 client.15513 fetch_fsmap learned FSMap version 25 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.191+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5583b7bfe7d0 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f50f3fff640 1 -- 172.21.15.64:0/228896728 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5104017af0 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f50f3fff640 1 -- 172.21.15.64:0/228896728 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f5110013070 con 0x5583b7be4990 2024-08-19T01:34:28.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f511ceae740 10 client.15513 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f511ceae740 10 client.15513 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f511ceae740 20 client.15513 populate_metadata read hostname 'smithi064' 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f511ceae740 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7bffc90 0x5583b7c02080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f511ceae740 4 client.15513 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583b7b581d0 con 0x5583b7bffc90 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.192+0000 7f510bfff640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7bffc90 0x5583b7c02080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.193+0000 7f50f3fff640 10 client.15513 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.193+0000 7f510bfff640 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7bffc90 0x5583b7c02080 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2d60cc63a6d101b server_cookie=8a037ab32ccd69da in_seq=0 out_seq=0 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.193+0000 7f50f3fff640 1 -- 172.21.15.64:0/228896728 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5583b7b581d0 con 0x5583b7bffc90 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.193+0000 7f50f3fff640 10 client.15513 handle_command_reply: tid=0 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.193+0000 7f511ceae740 1 client.15513 shutdown 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.193+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 msgr2=0x5583b7becae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.193+0000 7f511ceae740 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7becae0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5110009670 tx=0x7f51100047d0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 shutdown_connections 2024-08-19T01:34:28.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7bffc90 0x5583b7c02080 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7bed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be52b0 0x5583b7bed020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 --2- 172.21.15.64:0/228896728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7becae0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 >> 172.21.15.64:0/228896728 conn(0x5583b7be0f20 msgr2=0x5583b7be6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f50f3fff640 0 client.15513 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 shutdown_connections 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 1 -- 172.21.15.64:0/228896728 wait complete. 2024-08-19T01:34:28.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.194+0000 7f511ceae740 20 client.15513 trim_cache size 0 max 16384 2024-08-19T01:34:28.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 Processor -- start 2024-08-19T01:34:28.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 -- start start 2024-08-19T01:34:28.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7be73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7be52b0 0x5583b7c07900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7befb70 0x5583b7c07e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f51141b1710 con 0x5583b7be4990 2024-08-19T01:34:28.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f51141aefe0 con 0x5583b7be52b0 2024-08-19T01:34:28.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511ceae740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51141ace30 con 0x5583b7befb70 2024-08-19T01:34:28.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511904d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7befb70 0x5583b7c07e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511904d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7befb70 0x5583b7c07e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:37084/0 (socket says 172.21.15.64:37084) 2024-08-19T01:34:28.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.211+0000 7f511904d640 1 -- 172.21.15.64:0/1643702327 learned_addr learned my addr 172.21.15.64:0/1643702327 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:34:28.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f510bfff640 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7be52b0 0x5583b7c07900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f511884c640 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7be73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f511904d640 1 -- 172.21.15.64:0/1643702327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7be52b0 msgr2=0x5583b7c07900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f511904d640 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7be52b0 0x5583b7c07900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f511904d640 1 -- 172.21.15.64:0/1643702327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 msgr2=0x5583b7be73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f511904d640 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7be73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f511904d640 1 -- 172.21.15.64:0/1643702327 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b7be7930 con 0x5583b7befb70 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f511904d640 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7befb70 0x5583b7c07e40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f510c045000 tx=0x7f510c0b8fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dfb981c899246cc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f50fa7fc640 1 -- 172.21.15.64:0/1643702327 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c0c10c0 con 0x5583b7befb70 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f50fa7fc640 1 -- 172.21.15.64:0/1643702327 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f510c036030 con 0x5583b7befb70 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.212+0000 7f50fa7fc640 1 -- 172.21.15.64:0/1643702327 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c004b40 con 0x5583b7befb70 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 -- 172.21.15.64:0/1643702327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7befb70 msgr2=0x5583b7c07e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7befb70 0x5583b7c07e40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f510c045000 tx=0x7f510c0b8fb0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 -- 172.21.15.64:0/1643702327 shutdown_connections 2024-08-19T01:34:28.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7befb70 0x5583b7c07e40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7be52b0 0x5583b7c07900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 --2- 172.21.15.64:0/1643702327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be4990 0x5583b7be73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 -- 172.21.15.64:0/1643702327 >> 172.21.15.64:0/1643702327 conn(0x5583b7be0f20 msgr2=0x5583b7beeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:28.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 -- 172.21.15.64:0/1643702327 shutdown_connections 2024-08-19T01:34:28.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 -- 172.21.15.64:0/1643702327 wait complete. 2024-08-19T01:34:28.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.213+0000 7f511ceae740 1 -- start start 2024-08-19T01:34:28.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 1 Processor -- start 2024-08-19T01:34:28.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7bf7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bf8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7c63470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f51141ad4a0 con 0x5583b7be52b0 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f51141ae910 con 0x5583b7befb70 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51141adb70 con 0x5583b7be4990 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511ceae740 10 client.0 _ll_register_callbacks cb 0x7f511000dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f510bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bf8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f510bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bf8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:58142/0 (socket says 172.21.15.64:58142) 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f510bfff640 1 -- 172.21.15.64:0/1916061042 learned_addr learned my addr 172.21.15.64:0/1916061042 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511904d640 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7c63470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f511884c640 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7bf7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.214+0000 7f510bfff640 1 -- 172.21.15.64:0/1916061042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 msgr2=0x5583b7bf7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f510bfff640 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7bf7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f510bfff640 1 -- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 msgr2=0x5583b7c63470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f510bfff640 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7c63470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f510bfff640 1 -- 172.21.15.64:0/1916061042 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583b7c63b60 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f510bfff640 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bf8150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f510400c640 tx=0x7f510400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959884f04a21f14 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f50f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f50f3fff640 1 -- 172.21.15.64:0/1916061042 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104030070 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f50f3fff640 1 -- 172.21.15.64:0/1916061042 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5104006bf0 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f510c036030 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f50f3fff640 1 -- 172.21.15.64:0/1916061042 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f510403c050 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f511ceae740 10 client.15519 fetch_fsmap learned FSMap version 25 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5583b7bf5760 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.215+0000 7f50f3fff640 1 -- 172.21.15.64:0/1916061042 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f510c036030 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f50f3fff640 1 -- 172.21.15.64:0/1916061042 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f5104015040 con 0x5583b7be52b0 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511ceae740 10 client.15519 fetch_fsmap finished waiting for FSMap version 25 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511ceae740 10 client.15519 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:34:28.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511ceae740 20 client.15519 populate_metadata read hostname 'smithi064' 2024-08-19T01:34:28.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511ceae740 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7c73520 0x5583b7c75910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:34:28.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511ceae740 4 client.15519 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9fa8f803", "release": true, "await": true}] 2024-08-19T01:34:28.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9fa8f803", "release": true, "await": true}) -- 0x5583b7be1ad0 con 0x5583b7c73520 2024-08-19T01:34:28.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511884c640 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7c73520 0x5583b7c75910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:34:28.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f50f3fff640 10 client.15519 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:34:28.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.216+0000 7f511884c640 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7c73520 0x5583b7c75910 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9833e35e8c8fccd server_cookie=2423ca1342c0cadc in_seq=0 out_seq=0 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f50f3fff640 1 -- 172.21.15.64:0/1916061042 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5583b7be1ad0 con 0x5583b7c73520 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f50f3fff640 10 client.15519 handle_command_reply: tid=0 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 client.15519 shutdown 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 msgr2=0x5583b7bf8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bf8150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f510400c640 tx=0x7f510400c8b0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 shutdown_connections 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5583b7c73520 0x5583b7c75910 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f50f3fff640 0 client.15519 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5583b7befb70 0x5583b7c63470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5583b7be52b0 0x5583b7bf8150 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 --2- 172.21.15.64:0/1916061042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5583b7be4990 0x5583b7bf7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 >> 172.21.15.64:0/1916061042 conn(0x5583b7be0f20 msgr2=0x5583b7bdf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 shutdown_connections 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 1 -- 172.21.15.64:0/1916061042 wait complete. 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.220+0000 7f511ceae740 20 client.15519 trim_cache size 0 max 16384 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 25, 2024-08-19T01:34:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 4, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "9fa8f803": { 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 4, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 154.7, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:34:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:34:28.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f50fc0a2ea0 msgr2=0x7f50fc0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f50fc0a2ea0 0x7f50fc0a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5110009890 tx=0x7f5110008040 comp rx=0 tx=0).stop 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51141078b0 msgr2=0x7f511419cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51141078b0 0x7f511419cbb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f510c009ca0 tx=0x7f510c003ba0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 shutdown_connections 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f50fc0a2ea0 0x7f50fc0a5360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51141078b0 0x7f511419cbb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5114102250 0x7f5114198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 --2- 172.21.15.64:0/775810995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51141018b0 0x7f51141982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 >> 172.21.15.64:0/775810995 conn(0x7f51140fd5f0 msgr2=0x7f51140fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 shutdown_connections 2024-08-19T01:34:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:34:28.221+0000 7f511aab0640 1 -- 172.21.15.64:0/775810995 wait complete. 2024-08-19T01:34:28.227 INFO:tasks.workunit.client.0.smithi064.stderr:+ dbench 1 2024-08-19T01:34:28.273 INFO:tasks.workunit.client.0.smithi064.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-08-19T01:34:28.273 INFO:tasks.workunit.client.0.smithi064.stdout: 2024-08-19T01:34:28.273 INFO:tasks.workunit.client.0.smithi064.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-08-19T01:34:28.274 INFO:tasks.workunit.client.0.smithi064.stdout:0 of 1 processes prepared for launch 0 sec 2024-08-19T01:34:28.374 INFO:tasks.workunit.client.0.smithi064.stdout:1 of 1 processes prepared for launch 0 sec 2024-08-19T01:34:28.374 INFO:tasks.workunit.client.0.smithi064.stdout:releasing clients 2024-08-19T01:34:28.894 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9fa8f803', total seconds elapsed: 38.5 2024-08-19T01:34:29.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 3513 38.36 MB/sec warmup 1 sec latency 10.182 ms 2024-08-19T01:34:29.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:29 smithi064 ceph-mon[18515]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:34:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:29 smithi123 ceph-mon[21829]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:34:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:29 smithi161 ceph-mon[21757]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-08-19T01:34:30.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 6539 26.32 MB/sec warmup 2 sec latency 14.315 ms 2024-08-19T01:34:31.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 9538 22.11 MB/sec warmup 3 sec latency 12.697 ms 2024-08-19T01:34:31.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:31 smithi064 ceph-mon[18515]: pgmap v41: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 231 KiB/s wr, 33 op/s 2024-08-19T01:34:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:31 smithi123 ceph-mon[21829]: pgmap v41: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 231 KiB/s wr, 33 op/s 2024-08-19T01:34:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:31 smithi161 ceph-mon[21757]: pgmap v41: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 231 KiB/s wr, 33 op/s 2024-08-19T01:34:32.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 11812 20.09 MB/sec warmup 4 sec latency 64.024 ms 2024-08-19T01:34:32.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:32 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:34:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:32 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:34:33.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:32 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:34:33.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 14930 19.56 MB/sec warmup 5 sec latency 10.568 ms 2024-08-19T01:34:33.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:33 smithi064 ceph-mon[18515]: pgmap v42: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 231 KiB/s wr, 30 op/s 2024-08-19T01:34:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:33 smithi123 ceph-mon[21829]: pgmap v42: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 231 KiB/s wr, 30 op/s 2024-08-19T01:34:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:33 smithi161 ceph-mon[21757]: pgmap v42: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 231 KiB/s wr, 30 op/s 2024-08-19T01:34:34.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 18205 19.42 MB/sec warmup 6 sec latency 7.455 ms 2024-08-19T01:34:35.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 21566 19.35 MB/sec warmup 7 sec latency 20.813 ms 2024-08-19T01:34:35.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:35 smithi064 ceph-mon[18515]: pgmap v43: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 231 KiB/s wr, 30 op/s 2024-08-19T01:34:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:35 smithi123 ceph-mon[21829]: pgmap v43: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 231 KiB/s wr, 30 op/s 2024-08-19T01:34:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:35 smithi161 ceph-mon[21757]: pgmap v43: 129 pgs: 129 active+clean; 3.7 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 231 KiB/s wr, 30 op/s 2024-08-19T01:34:36.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 24869 19.15 MB/sec warmup 8 sec latency 10.644 ms 2024-08-19T01:34:37.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 27864 18.60 MB/sec warmup 9 sec latency 11.203 ms 2024-08-19T01:34:37.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:37 smithi064 ceph-mon[18515]: pgmap v44: 129 pgs: 129 active+clean; 11 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1002 KiB/s wr, 122 op/s 2024-08-19T01:34:37.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:37 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:34:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:37 smithi123 ceph-mon[21829]: pgmap v44: 129 pgs: 129 active+clean; 11 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1002 KiB/s wr, 122 op/s 2024-08-19T01:34:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:37 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:34:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:37 smithi161 ceph-mon[21757]: pgmap v44: 129 pgs: 129 active+clean; 11 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1002 KiB/s wr, 122 op/s 2024-08-19T01:34:38.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:37 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:34:38.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 30795 18.10 MB/sec warmup 10 sec latency 46.527 ms 2024-08-19T01:34:39.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 33632 18.03 MB/sec warmup 11 sec latency 10.851 ms 2024-08-19T01:34:39.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:39 smithi064 ceph-mon[18515]: pgmap v45: 129 pgs: 129 active+clean; 11 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1002 KiB/s wr, 119 op/s 2024-08-19T01:34:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:39 smithi123 ceph-mon[21829]: pgmap v45: 129 pgs: 129 active+clean; 11 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1002 KiB/s wr, 119 op/s 2024-08-19T01:34:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:39 smithi161 ceph-mon[21757]: pgmap v45: 129 pgs: 129 active+clean; 11 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1002 KiB/s wr, 119 op/s 2024-08-19T01:34:40.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 36692 17.86 MB/sec warmup 12 sec latency 12.640 ms 2024-08-19T01:34:41.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 39778 17.81 MB/sec warmup 13 sec latency 22.580 ms 2024-08-19T01:34:41.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:41 smithi064 ceph-mon[18515]: pgmap v46: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.7 MiB/s wr, 213 op/s 2024-08-19T01:34:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:41 smithi123 ceph-mon[21829]: pgmap v46: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.7 MiB/s wr, 213 op/s 2024-08-19T01:34:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:41 smithi161 ceph-mon[21757]: pgmap v46: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.7 MiB/s wr, 213 op/s 2024-08-19T01:34:42.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 43073 17.89 MB/sec warmup 14 sec latency 8.607 ms 2024-08-19T01:34:43.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:42 smithi123 ceph-mon[21829]: pgmap v47: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-08-19T01:34:43.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:42 smithi161 ceph-mon[21757]: pgmap v47: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-08-19T01:34:43.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:42 smithi064 ceph-mon[18515]: pgmap v47: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-08-19T01:34:43.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 45955 17.66 MB/sec warmup 15 sec latency 168.587 ms 2024-08-19T01:34:44.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 48755 17.35 MB/sec warmup 16 sec latency 10.203 ms 2024-08-19T01:34:45.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:44 smithi123 ceph-mon[21829]: pgmap v48: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-08-19T01:34:45.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:44 smithi161 ceph-mon[21757]: pgmap v48: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-08-19T01:34:45.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:44 smithi064 ceph-mon[18515]: pgmap v48: 129 pgs: 129 active+clean; 18 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-08-19T01:34:45.374 INFO:tasks.workunit.client.0.smithi064.stdout: 1 51670 17.30 MB/sec warmup 17 sec latency 17.114 ms 2024-08-19T01:34:46.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 54425 17.22 MB/sec warmup 18 sec latency 15.268 ms 2024-08-19T01:34:47.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:46 smithi161 ceph-mon[21757]: pgmap v49: 129 pgs: 129 active+clean; 25 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-08-19T01:34:47.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:46 smithi161 ceph-mon[21757]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-08-19T01:34:47.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:46 smithi064 ceph-mon[18515]: pgmap v49: 129 pgs: 129 active+clean; 25 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-08-19T01:34:47.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:46 smithi064 ceph-mon[18515]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-08-19T01:34:47.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 57654 17.24 MB/sec warmup 19 sec latency 17.286 ms 2024-08-19T01:34:47.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:46 smithi123 ceph-mon[21829]: pgmap v49: 129 pgs: 129 active+clean; 25 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-08-19T01:34:47.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:46 smithi123 ceph-mon[21829]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-08-19T01:34:48.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 60987 17.31 MB/sec warmup 20 sec latency 19.506 ms 2024-08-19T01:34:49.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:48 smithi161 ceph-mon[21757]: pgmap v50: 129 pgs: 129 active+clean; 25 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:49.217 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:48 smithi123 ceph-mon[21829]: pgmap v50: 129 pgs: 129 active+clean; 25 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:49.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:48 smithi064 ceph-mon[18515]: pgmap v50: 129 pgs: 129 active+clean; 25 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:49.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 64319 17.39 MB/sec warmup 21 sec latency 10.955 ms 2024-08-19T01:34:50.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 67629 17.45 MB/sec warmup 22 sec latency 19.533 ms 2024-08-19T01:34:51.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:50 smithi161 ceph-mon[21757]: pgmap v51: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 276 op/s 2024-08-19T01:34:51.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:50 smithi064 ceph-mon[18515]: pgmap v51: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 276 op/s 2024-08-19T01:34:51.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 70781 17.34 MB/sec warmup 23 sec latency 10.455 ms 2024-08-19T01:34:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:50 smithi123 ceph-mon[21829]: pgmap v51: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 276 op/s 2024-08-19T01:34:52.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 73554 17.14 MB/sec warmup 24 sec latency 12.749 ms 2024-08-19T01:34:53.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:52 smithi161 ceph-mon[21757]: pgmap v52: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:53.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:52 smithi064 ceph-mon[18515]: pgmap v52: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:53.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 76845 17.18 MB/sec warmup 25 sec latency 11.312 ms 2024-08-19T01:34:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:52 smithi123 ceph-mon[21829]: pgmap v52: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:54.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 79270 17.07 MB/sec warmup 26 sec latency 15.807 ms 2024-08-19T01:34:55.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:54 smithi161 ceph-mon[21757]: pgmap v53: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:55.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:54 smithi064 ceph-mon[18515]: pgmap v53: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:55.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 80537 16.62 MB/sec warmup 27 sec latency 346.049 ms 2024-08-19T01:34:55.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:54 smithi123 ceph-mon[21829]: pgmap v53: 129 pgs: 129 active+clean; 33 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-08-19T01:34:56.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 82387 16.47 MB/sec warmup 28 sec latency 23.429 ms 2024-08-19T01:34:57.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:56 smithi161 ceph-mon[21757]: pgmap v54: 129 pgs: 129 active+clean; 118 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.9 MiB/s wr, 364 op/s 2024-08-19T01:34:57.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:56 smithi064 ceph-mon[18515]: pgmap v54: 129 pgs: 129 active+clean; 118 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.9 MiB/s wr, 364 op/s 2024-08-19T01:34:57.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 85488 16.54 MB/sec warmup 29 sec latency 12.411 ms 2024-08-19T01:34:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:56 smithi123 ceph-mon[21829]: pgmap v54: 129 pgs: 129 active+clean; 118 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.9 MiB/s wr, 364 op/s 2024-08-19T01:34:58.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 88675 16.58 MB/sec warmup 30 sec latency 12.686 ms 2024-08-19T01:34:59.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:34:58 smithi161 ceph-mon[21757]: pgmap v55: 129 pgs: 129 active+clean; 118 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 8.1 MiB/s wr, 276 op/s 2024-08-19T01:34:59.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:34:58 smithi064 ceph-mon[18515]: pgmap v55: 129 pgs: 129 active+clean; 118 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 8.1 MiB/s wr, 276 op/s 2024-08-19T01:34:59.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 91285 16.36 MB/sec warmup 31 sec latency 10.529 ms 2024-08-19T01:34:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:34:58 smithi123 ceph-mon[21829]: pgmap v55: 129 pgs: 129 active+clean; 118 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 8.1 MiB/s wr, 276 op/s 2024-08-19T01:35:00.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 93775 16.35 MB/sec warmup 32 sec latency 11.656 ms 2024-08-19T01:35:01.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:00 smithi161 ceph-mon[21757]: pgmap v56: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.0 MiB/s wr, 455 op/s 2024-08-19T01:35:01.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:00 smithi064 ceph-mon[18515]: pgmap v56: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.0 MiB/s wr, 455 op/s 2024-08-19T01:35:01.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 96008 16.26 MB/sec warmup 33 sec latency 51.985 ms 2024-08-19T01:35:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:00 smithi123 ceph-mon[21829]: pgmap v56: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.0 MiB/s wr, 455 op/s 2024-08-19T01:35:02.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 98113 16.00 MB/sec warmup 34 sec latency 47.894 ms 2024-08-19T01:35:03.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:02 smithi161 ceph-mon[21757]: pgmap v57: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.2 MiB/s wr, 361 op/s 2024-08-19T01:35:03.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:02 smithi064 ceph-mon[18515]: pgmap v57: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.2 MiB/s wr, 361 op/s 2024-08-19T01:35:03.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 100465 15.96 MB/sec warmup 35 sec latency 19.709 ms 2024-08-19T01:35:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:02 smithi123 ceph-mon[21829]: pgmap v57: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.2 MiB/s wr, 361 op/s 2024-08-19T01:35:04.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 103588 15.99 MB/sec warmup 36 sec latency 11.327 ms 2024-08-19T01:35:05.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:04 smithi161 ceph-mon[21757]: pgmap v58: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.2 MiB/s wr, 361 op/s 2024-08-19T01:35:05.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:04 smithi064 ceph-mon[18515]: pgmap v58: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.2 MiB/s wr, 361 op/s 2024-08-19T01:35:05.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 106827 16.07 MB/sec warmup 37 sec latency 10.884 ms 2024-08-19T01:35:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:04 smithi123 ceph-mon[21829]: pgmap v58: 129 pgs: 129 active+clean; 54 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.2 MiB/s wr, 361 op/s 2024-08-19T01:35:06.375 INFO:tasks.workunit.client.0.smithi064.stdout: 1 109965 16.11 MB/sec warmup 38 sec latency 18.757 ms 2024-08-19T01:35:07.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:06 smithi161 ceph-mon[21757]: pgmap v59: 129 pgs: 129 active+clean; 62 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.1 MiB/s wr, 477 op/s 2024-08-19T01:35:07.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:06 smithi064 ceph-mon[18515]: pgmap v59: 129 pgs: 129 active+clean; 62 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.1 MiB/s wr, 477 op/s 2024-08-19T01:35:07.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 112595 15.95 MB/sec warmup 39 sec latency 14.135 ms 2024-08-19T01:35:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:06 smithi123 ceph-mon[21829]: pgmap v59: 129 pgs: 129 active+clean; 62 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.1 MiB/s wr, 477 op/s 2024-08-19T01:35:08.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 114946 15.95 MB/sec warmup 40 sec latency 40.476 ms 2024-08-19T01:35:09.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:08 smithi161 ceph-mon[21757]: pgmap v60: 129 pgs: 129 active+clean; 62 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 295 op/s 2024-08-19T01:35:09.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:08 smithi064 ceph-mon[18515]: pgmap v60: 129 pgs: 129 active+clean; 62 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 295 op/s 2024-08-19T01:35:09.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 117787 15.91 MB/sec warmup 41 sec latency 18.136 ms 2024-08-19T01:35:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:08 smithi123 ceph-mon[21829]: pgmap v60: 129 pgs: 129 active+clean; 62 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 295 op/s 2024-08-19T01:35:10.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 120962 15.97 MB/sec warmup 42 sec latency 13.823 ms 2024-08-19T01:35:11.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:10 smithi064 ceph-mon[18515]: pgmap v61: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.5 MiB/s wr, 391 op/s 2024-08-19T01:35:11.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 124168 16.01 MB/sec warmup 43 sec latency 13.344 ms 2024-08-19T01:35:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:10 smithi123 ceph-mon[21829]: pgmap v61: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.5 MiB/s wr, 391 op/s 2024-08-19T01:35:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:10 smithi161 ceph-mon[21757]: pgmap v61: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.5 MiB/s wr, 391 op/s 2024-08-19T01:35:12.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 126677 15.87 MB/sec warmup 44 sec latency 11.170 ms 2024-08-19T01:35:13.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:12 smithi064 ceph-mon[18515]: pgmap v62: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-08-19T01:35:13.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 128987 15.81 MB/sec warmup 45 sec latency 20.133 ms 2024-08-19T01:35:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:12 smithi123 ceph-mon[21829]: pgmap v62: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-08-19T01:35:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:12 smithi161 ceph-mon[21757]: pgmap v62: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-08-19T01:35:14.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 131999 15.84 MB/sec warmup 46 sec latency 15.909 ms 2024-08-19T01:35:15.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:14 smithi064 ceph-mon[18515]: pgmap v63: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-08-19T01:35:15.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 135146 15.90 MB/sec warmup 47 sec latency 12.601 ms 2024-08-19T01:35:15.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:14 smithi123 ceph-mon[21829]: pgmap v63: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-08-19T01:35:15.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:14 smithi161 ceph-mon[21757]: pgmap v63: 129 pgs: 129 active+clean; 69 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 211 op/s 2024-08-19T01:35:16.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 138326 15.94 MB/sec warmup 48 sec latency 11.190 ms 2024-08-19T01:35:17.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:16 smithi064 ceph-mon[18515]: pgmap v64: 129 pgs: 129 active+clean; 79 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-08-19T01:35:17.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 140938 15.81 MB/sec warmup 49 sec latency 15.789 ms 2024-08-19T01:35:17.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:16 smithi123 ceph-mon[21829]: pgmap v64: 129 pgs: 129 active+clean; 79 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-08-19T01:35:17.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:16 smithi161 ceph-mon[21757]: pgmap v64: 129 pgs: 129 active+clean; 79 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-08-19T01:35:18.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 143441 15.82 MB/sec warmup 50 sec latency 15.614 ms 2024-08-19T01:35:19.217 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:18 smithi123 ceph-mon[21829]: pgmap v65: 129 pgs: 129 active+clean; 79 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 193 op/s 2024-08-19T01:35:19.217 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:18 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:35:19.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:18 smithi064 ceph-mon[18515]: pgmap v65: 129 pgs: 129 active+clean; 79 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 193 op/s 2024-08-19T01:35:19.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:18 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:35:19.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 146141 15.79 MB/sec warmup 51 sec latency 39.032 ms 2024-08-19T01:35:19.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:18 smithi161 ceph-mon[21757]: pgmap v65: 129 pgs: 129 active+clean; 79 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 193 op/s 2024-08-19T01:35:19.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:18 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:35:20.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 149377 15.85 MB/sec warmup 52 sec latency 11.239 ms 2024-08-19T01:35:20.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:35:20.416 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:19 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:35:20.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:19 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:20.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:35:20.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:35:20.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:19 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:35:21.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 152557 15.88 MB/sec warmup 53 sec latency 11.190 ms 2024-08-19T01:35:21.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:21 smithi064 ceph-mon[18515]: pgmap v66: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-08-19T01:35:21.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:21 smithi123 ceph-mon[21829]: pgmap v66: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-08-19T01:35:21.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:21 smithi161 ceph-mon[21757]: pgmap v66: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-08-19T01:35:22.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 155004 15.75 MB/sec warmup 54 sec latency 13.965 ms 2024-08-19T01:35:23.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 157503 15.77 MB/sec warmup 55 sec latency 17.236 ms 2024-08-19T01:35:23.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:23 smithi064 ceph-mon[18515]: pgmap v67: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-08-19T01:35:23.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:23 smithi123 ceph-mon[21829]: pgmap v67: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-08-19T01:35:23.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:23 smithi161 ceph-mon[21757]: pgmap v67: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-08-19T01:35:24.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 160379 15.75 MB/sec warmup 56 sec latency 14.722 ms 2024-08-19T01:35:25.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 163559 15.80 MB/sec warmup 57 sec latency 6.571 ms 2024-08-19T01:35:25.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:25 smithi064 ceph-mon[18515]: pgmap v68: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-08-19T01:35:25.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:25 smithi123 ceph-mon[21829]: pgmap v68: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-08-19T01:35:25.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:25 smithi161 ceph-mon[21757]: pgmap v68: 129 pgs: 129 active+clean; 87 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-08-19T01:35:26.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 166531 15.78 MB/sec warmup 58 sec latency 13.976 ms 2024-08-19T01:35:27.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 169122 15.71 MB/sec warmup 59 sec latency 15.895 ms 2024-08-19T01:35:27.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:27 smithi064 ceph-mon[18515]: pgmap v69: 129 pgs: 129 active+clean; 95 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-08-19T01:35:27.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:27 smithi123 ceph-mon[21829]: pgmap v69: 129 pgs: 129 active+clean; 95 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-08-19T01:35:27.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:27 smithi161 ceph-mon[21757]: pgmap v69: 129 pgs: 129 active+clean; 95 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-08-19T01:35:28.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 171494 15.68 MB/sec warmup 60 sec latency 10.850 ms 2024-08-19T01:35:29.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 174557 15.71 MB/sec warmup 61 sec latency 14.856 ms 2024-08-19T01:35:29.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:29 smithi064 ceph-mon[18515]: pgmap v70: 129 pgs: 129 active+clean; 95 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-08-19T01:35:29.750 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:29 smithi161 ceph-mon[21757]: pgmap v70: 129 pgs: 129 active+clean; 95 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-08-19T01:35:29.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:29 smithi123 ceph-mon[21829]: pgmap v70: 129 pgs: 129 active+clean; 95 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-08-19T01:35:30.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 177482 15.74 MB/sec warmup 62 sec latency 46.059 ms 2024-08-19T01:35:31.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 179795 15.62 MB/sec warmup 63 sec latency 35.796 ms 2024-08-19T01:35:31.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:31 smithi123 ceph-mon[21829]: pgmap v71: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-08-19T01:35:31.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:31 smithi161 ceph-mon[21757]: pgmap v71: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-08-19T01:35:31.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:31 smithi064 ceph-mon[18515]: pgmap v71: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-08-19T01:35:32.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 181892 15.58 MB/sec warmup 64 sec latency 86.352 ms 2024-08-19T01:35:33.338 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-08-19T01:35:33.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:35:33.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 185025 15.62 MB/sec warmup 65 sec latency 12.879 ms 2024-08-19T01:35:33.638 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:35:33.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:33 smithi064 ceph-mon[18515]: pgmap v72: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-08-19T01:35:33.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:33 smithi123 ceph-mon[21829]: pgmap v72: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-08-19T01:35:33.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:33 smithi161 ceph-mon[21757]: pgmap v72: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 -- 172.21.15.64:0/1155211499 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b0100b50 msgr2=0x7f36b0100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 --2- 172.21.15.64:0/1155211499 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b0100b50 0x7f36b0100f30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009900 tx=0x7f36a002a480 comp rx=0 tx=0).stop 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 -- 172.21.15.64:0/1155211499 shutdown_connections 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 --2- 172.21.15.64:0/1155211499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b0102c20 0x7f36b010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 --2- 172.21.15.64:0/1155211499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b01022a0 0x7f36b01026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 --2- 172.21.15.64:0/1155211499 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b0100b50 0x7f36b0100f30 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 -- 172.21.15.64:0/1155211499 >> 172.21.15.64:0/1155211499 conn(0x7f36b00fc500 msgr2=0x7f36b00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:34.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 -- 172.21.15.64:0/1155211499 shutdown_connections 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 -- 172.21.15.64:0/1155211499 wait complete. 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.191+0000 7f36b614a640 1 Processor -- start 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36b614a640 1 -- start start 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36b614a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b0100b50 0x7f36b00fc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36b614a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b01022a0 0x7f36b00fcf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36b614a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b0102c20 0x7f36b00fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36b614a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f36b0110d00 con 0x7f36b01022a0 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36b614a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f36b0110b80 con 0x7f36b0100b50 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36b614a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f36b0110e80 con 0x7f36b0102c20 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36af7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b0100b50 0x7f36b00fc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36af7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b0100b50 0x7f36b00fc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:59000/0 (socket says 172.21.15.64:59000) 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36af7fe640 1 -- 172.21.15.64:0/1274840274 learned_addr learned my addr 172.21.15.64:0/1274840274 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:34.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36affff640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b0102c20 0x7f36b00fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.192+0000 7f36a7fff640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b01022a0 0x7f36b00fcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36a7fff640 1 -- 172.21.15.64:0/1274840274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b0102c20 msgr2=0x7f36b00fd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36a7fff640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b0102c20 0x7f36b00fd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36a7fff640 1 -- 172.21.15.64:0/1274840274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b0100b50 msgr2=0x7f36b00fc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36a7fff640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b0100b50 0x7f36b00fc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36a7fff640 1 -- 172.21.15.64:0/1274840274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b01aab80 con 0x7f36b01022a0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36af7fe640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b0100b50 0x7f36b00fc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36a7fff640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b01022a0 0x7f36b00fcf30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f369800e9f0 tx=0x7f369800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36ad7fa640 1 -- 172.21.15.64:0/1274840274 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369800ccc0 con 0x7f36b01022a0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.193+0000 7f36ad7fa640 1 -- 172.21.15.64:0/1274840274 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3698010910 con 0x7f36b01022a0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.194+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b01aae10 con 0x7f36b01022a0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.194+0000 7f36ad7fa640 1 -- 172.21.15.64:0/1274840274 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698018960 con 0x7f36b01022a0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.194+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b01ab310 con 0x7f36b01022a0 2024-08-19T01:35:34.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.195+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b0101ee0 con 0x7f36b01022a0 2024-08-19T01:35:34.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.196+0000 7f36ad7fa640 1 -- 172.21.15.64:0/1274840274 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3698010ac0 con 0x7f36b01022a0 2024-08-19T01:35:34.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.196+0000 7f36ad7fa640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f369c0a2d70 0x7f369c0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:34.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.196+0000 7f36af7fe640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f369c0a2d70 0x7f369c0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:34.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.196+0000 7f36ad7fa640 1 -- 172.21.15.64:0/1274840274 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f36980cfb10 con 0x7f36b01022a0 2024-08-19T01:35:34.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.197+0000 7f36af7fe640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f369c0a2d70 0x7f369c0a5230 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f36a0000c00 tx=0x7f36a00046c0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:34.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.202+0000 7f36ad7fa640 1 -- 172.21.15.64:0/1274840274 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f369809bd30 con 0x7f36b01022a0 2024-08-19T01:35:34.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 188147 15.65 MB/sec warmup 66 sec latency 12.979 ms 2024-08-19T01:35:34.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.443+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36b002d5e0 con 0x7f36b01022a0 2024-08-19T01:35:34.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.444+0000 7f36ad7fa640 1 -- 172.21.15.64:0/1274840274 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11924 (secure 0 0 0) 0x7f36980a0030 con 0x7f36b01022a0 2024-08-19T01:35:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:35:34.445 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"btime":"2024-08-19T01:34:36:714408+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:34:36.547976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:35:34.445 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2024-08-19T01:35:34.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f369c0a2d70 msgr2=0x7f369c0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:34.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f369c0a2d70 0x7f369c0a5230 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f36a0000c00 tx=0x7f36a00046c0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b01022a0 msgr2=0x7f36b00fcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b01022a0 0x7f36b00fcf30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f369800e9f0 tx=0x7f369800eeb0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 shutdown_connections 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f369c0a2d70 0x7f369c0a5230 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b0102c20 0x7f36b00fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36b01022a0 0x7f36b00fcf30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 --2- 172.21.15.64:0/1274840274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36b0100b50 0x7f36b00fc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 >> 172.21.15.64:0/1274840274 conn(0x7f36b00fc500 msgr2=0x7f36b0069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 shutdown_connections 2024-08-19T01:35:34.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:34.446+0000 7f36b614a640 1 -- 172.21.15.64:0/1274840274 wait complete. 2024-08-19T01:35:34.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:34 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1274840274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:35:34.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:34 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1274840274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:35:34.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:34 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1274840274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:35:35.198 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-08-19T01:35:35.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-08-19T01:35:35.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 190602 15.56 MB/sec warmup 67 sec latency 77.808 ms 2024-08-19T01:35:35.532 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:35:35.773 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:35 smithi064 ceph-mon[18515]: pgmap v73: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-08-19T01:35:35.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:35 smithi123 ceph-mon[21829]: pgmap v73: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-08-19T01:35:35.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:35 smithi161 ceph-mon[21757]: pgmap v73: 129 pgs: 129 active+clean; 102 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-08-19T01:35:36.376 INFO:tasks.workunit.client.0.smithi064.stdout: 1 192606 15.52 MB/sec warmup 68 sec latency 78.414 ms 2024-08-19T01:35:36.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 -- 172.21.15.64:0/2848996354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538103470 msgr2=0x7fe5381078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/2848996354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538103470 0x7fe5381078b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009930 tx=0x7fe52c02a420 comp rx=0 tx=0).stop 2024-08-19T01:35:36.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 -- 172.21.15.64:0/2848996354 shutdown_connections 2024-08-19T01:35:36.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/2848996354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe538107fc0 0x7fe53810e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/2848996354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538103470 0x7fe5381078b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/2848996354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538102ac0 0x7fe538102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 -- 172.21.15.64:0/2848996354 >> 172.21.15.64:0/2848996354 conn(0x7fe5380fc500 msgr2=0x7fe5380fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 -- 172.21.15.64:0/2848996354 shutdown_connections 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.529+0000 7fe53e4c4640 1 -- 172.21.15.64:0/2848996354 wait complete. 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 Processor -- start 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 -- start start 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538102ac0 0x7fe53819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538103470 0x7fe53819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe538107fc0 0x7fe53819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe538111dc0 con 0x7fe538102ac0 2024-08-19T01:35:36.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe538111c40 con 0x7fe538103470 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53e4c4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe538111f40 con 0x7fe538107fc0 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe537fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538102ac0 0x7fe53819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe537fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538102ac0 0x7fe53819a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59002/0 (socket says 172.21.15.64:59002) 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe5377fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538103470 0x7fe53819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe537fff640 1 -- 172.21.15.64:0/1567295798 learned_addr learned my addr 172.21.15.64:0/1567295798 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.530+0000 7fe53ca61640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe538107fc0 0x7fe53819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe537fff640 1 -- 172.21.15.64:0/1567295798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe538107fc0 msgr2=0x7fe53819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe537fff640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe538107fc0 0x7fe53819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe537fff640 1 -- 172.21.15.64:0/1567295798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538103470 msgr2=0x7fe53819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe537fff640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538103470 0x7fe53819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe537fff640 1 -- 172.21.15.64:0/1567295798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53819f4b0 con 0x7fe538102ac0 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe5377fe640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538103470 0x7fe53819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:35:36.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe537fff640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538102ac0 0x7fe53819a500 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ea60 tx=0x7fe52800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:36.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe5357fa640 1 -- 172.21.15.64:0/1567295798 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52800cde0 con 0x7fe538102ac0 2024-08-19T01:35:36.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53819f7a0 con 0x7fe538102ac0 2024-08-19T01:35:36.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe5357fa640 1 -- 172.21.15.64:0/1567295798 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5280035f0 con 0x7fe538102ac0 2024-08-19T01:35:36.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe5357fa640 1 -- 172.21.15.64:0/1567295798 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe528010670 con 0x7fe538102ac0 2024-08-19T01:35:36.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.531+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5381a7080 con 0x7fe538102ac0 2024-08-19T01:35:36.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.533+0000 7fe5357fa640 1 -- 172.21.15.64:0/1567295798 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe5280037a0 con 0x7fe538102ac0 2024-08-19T01:35:36.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 Processor -- start 2024-08-19T01:35:36.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 -- start start 2024-08-19T01:35:36.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5357fa640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5200a2cf0 0x7fe5200a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868279510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279a50 0x55a868283dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868284310 0x55a868286700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a8681dea70 con 0x55a868284310 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55a868176780 con 0x55a868279130 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5408c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a8681daa30 con 0x55a868279a50 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe53ca61640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5200a2cf0 0x7fe5200a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe537fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868279510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe5377fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279a50 0x55a868283dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe537fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868279510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:59046/0 (socket says 172.21.15.64:59046) 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe537fff640 1 -- 172.21.15.64:0/4160964629 learned_addr learned my addr 172.21.15.64:0/4160964629 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.534+0000 7fe53ca61640 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868284310 0x55a868286700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe5357fa640 1 -- 172.21.15.64:0/1567295798 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe5280cfaf0 con 0x7fe538102ac0 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe537fff640 1 -- 172.21.15.64:0/4160964629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279a50 msgr2=0x55a868283dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe537fff640 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279a50 0x55a868283dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe537fff640 1 -- 172.21.15.64:0/4160964629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868284310 msgr2=0x55a868286700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe537fff640 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868284310 0x55a868286700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe537fff640 1 -- 172.21.15.64:0/4160964629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a867f50b90 con 0x55a868279130 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe537fff640 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868279510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe528098630 tx=0x7fe5280057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf6d5a68eae4494 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe53ca61640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5200a2cf0 0x7fe5200a51b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe530005fd0 tx=0x7fe530009e90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe51e7fc640 1 -- 172.21.15.64:0/4160964629 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5280b4e60 con 0x55a868279130 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe51e7fc640 1 -- 172.21.15.64:0/4160964629 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5280b1040 con 0x55a868279130 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.535+0000 7fe51e7fc640 1 -- 172.21.15.64:0/4160964629 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52809ca70 con 0x55a868279130 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 -- 172.21.15.64:0/4160964629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 msgr2=0x55a868279510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868279510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe528098630 tx=0x7fe5280057d0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 -- 172.21.15.64:0/4160964629 shutdown_connections 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868284310 0x55a868286700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279a50 0x55a868283dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 --2- 172.21.15.64:0/4160964629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868279510 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 -- 172.21.15.64:0/4160964629 >> 172.21.15.64:0/4160964629 conn(0x55a8682756c0 msgr2=0x55a868275aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 -- 172.21.15.64:0/4160964629 shutdown_connections 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.536+0000 7fe5408c2740 1 -- 172.21.15.64:0/4160964629 wait complete. 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 -- start start 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 Processor -- start 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868281770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868281cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a868286ee0 con 0x55a868279a50 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55a868199990 con 0x55a868279130 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.537+0000 7fe5408c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a868287060 con 0x55a868284310 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe537fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe53ca61640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868281cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe537fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:59060/0 (socket says 172.21.15.64:59060) 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe537fff640 1 -- 172.21.15.64:0/2643579589 learned_addr learned my addr 172.21.15.64:0/2643579589 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe5408c2740 10 client.0 _ll_register_callbacks cb 0x55a867eee090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe5377fe640 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868281770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe53ca61640 1 -- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 msgr2=0x55a868281230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe53ca61640 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe53ca61640 1 -- 172.21.15.64:0/2643579589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 msgr2=0x55a868281770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe53ca61640 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868281770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe53ca61640 1 -- 172.21.15.64:0/2643579589 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8681986c0 con 0x55a868284310 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe537fff640 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe53ca61640 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868281cb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe53000f850 tx=0x7fe53001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd34649980a27023 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.538+0000 7fe5177fe640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.539+0000 7fe5177fe640 1 -- 172.21.15.64:0/2643579589 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530024070 con 0x55a868284310 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.539+0000 7fe5177fe640 1 -- 172.21.15.64:0/2643579589 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe530004690 con 0x55a868284310 2024-08-19T01:35:36.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.539+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5280b1040 con 0x55a868284310 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.539+0000 7fe5177fe640 1 -- 172.21.15.64:0/2643579589 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe530033050 con 0x55a868284310 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.539+0000 7fe5408c2740 10 client.25003 fetch_fsmap learned FSMap version 27 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.539+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a868292bc0 con 0x55a868284310 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5177fe640 1 -- 172.21.15.64:0/2643579589 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe53001f930 con 0x55a868284310 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5177fe640 1 -- 172.21.15.64:0/2643579589 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe530028500 con 0x55a868284310 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5408c2740 10 client.25003 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5408c2740 10 client.25003 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5408c2740 20 client.25003 populate_metadata read hostname 'smithi064' 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868296500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5408c2740 4 client.25003 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe5280b1040 con 0x55a8682940d0 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe537fff640 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868296500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe537fff640 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868296500 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8f4092bd73ffc38 server_cookie=7db66e77b1451fe2 in_seq=0 out_seq=0 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.540+0000 7fe5177fe640 10 client.25003 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5177fe640 1 -- 172.21.15.64:0/2643579589 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fe5280b1040 con 0x55a8682940d0 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5177fe640 10 client.25003 handle_command_reply: tid=0 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 client.25003 shutdown 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 msgr2=0x55a868281cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868281cb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe53000f850 tx=0x7fe53001d000 comp rx=0 tx=0).stop 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 shutdown_connections 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868281cb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868281770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868296500 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5177fe640 0 client.25003 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2643579589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 >> 172.21.15.64:0/2643579589 conn(0x55a8682756c0 msgr2=0x55a86827b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.541+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 shutdown_connections 2024-08-19T01:35:36.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.542+0000 7fe5408c2740 1 -- 172.21.15.64:0/2643579589 wait complete. 2024-08-19T01:35:36.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.542+0000 7fe5408c2740 20 client.25003 trim_cache size 0 max 16384 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 Processor -- start 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 -- start start 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a8682745e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868274b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe5381b3950 con 0x55a868279a50 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5381b1220 con 0x55a868279130 2024-08-19T01:35:36.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.559+0000 7fe5408c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5381af070 con 0x55a868284310 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe5377fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a8682745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe53ca61640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868274b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe537fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe5377fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a8682745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59030/0 (socket says 172.21.15.64:59030) 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe5377fe640 1 -- 172.21.15.64:0/2681326250 learned_addr learned my addr 172.21.15.64:0/2681326250 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe537fff640 1 -- 172.21.15.64:0/2681326250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 msgr2=0x55a868274b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe537fff640 1 --2- 172.21.15.64:0/2681326250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868274b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe537fff640 1 -- 172.21.15.64:0/2681326250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 msgr2=0x55a8682745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe537fff640 1 --2- 172.21.15.64:0/2681326250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a8682745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe537fff640 1 -- 172.21.15.64:0/2681326250 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a868273c70 con 0x55a868279130 2024-08-19T01:35:36.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe537fff640 1 --2- 172.21.15.64:0/2681326250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281b00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe528000c00 tx=0x7fe52809c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26270ca302c1ba8d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:36.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe51dffb640 1 -- 172.21.15.64:0/2681326250 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5280b4e60 con 0x55a868279130 2024-08-19T01:35:36.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe51dffb640 1 -- 172.21.15.64:0/2681326250 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5280b1040 con 0x55a868279130 2024-08-19T01:35:36.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.560+0000 7fe51dffb640 1 -- 172.21.15.64:0/2681326250 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5280a1430 con 0x55a868279130 2024-08-19T01:35:36.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 -- 172.21.15.64:0/2681326250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 msgr2=0x55a868281b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2681326250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281b00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe528000c00 tx=0x7fe52809c5f0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 -- 172.21.15.64:0/2681326250 shutdown_connections 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2681326250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868284310 0x55a868274b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2681326250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a8682745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 --2- 172.21.15.64:0/2681326250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868279130 0x55a868281b00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 -- 172.21.15.64:0/2681326250 >> 172.21.15.64:0/2681326250 conn(0x55a8682756c0 msgr2=0x55a868273ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 -- 172.21.15.64:0/2681326250 shutdown_connections 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 -- 172.21.15.64:0/2681326250 wait complete. 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.561+0000 7fe5408c2740 1 -- start start 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 1 Processor -- start 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279130 0x55a86828ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868287d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868284310 0x55a8682882c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe5381af6e0 con 0x55a868279a50 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5381b0b50 con 0x55a868284310 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5381afdb0 con 0x55a868279130 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5408c2740 10 client.0 _ll_register_callbacks cb 0x7fe530009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe537fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279130 0x55a86828ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe5377fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868287d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe53ca61640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868284310 0x55a8682882c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe537fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279130 0x55a86828ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:50514/0 (socket says 172.21.15.64:50514) 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.562+0000 7fe537fff640 1 -- 172.21.15.64:0/3810568528 learned_addr learned my addr 172.21.15.64:0/3810568528 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe53ca61640 1 -- 172.21.15.64:0/3810568528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279130 msgr2=0x55a86828ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe53ca61640 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279130 0x55a86828ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe53ca61640 1 -- 172.21.15.64:0/3810568528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 msgr2=0x55a868287d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe53ca61640 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868287d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe53ca61640 1 -- 172.21.15.64:0/3810568528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a868288af0 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5377fe640 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868287d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe53ca61640 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868284310 0x55a8682882c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe530009bb0 tx=0x7fe530028d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4671b6968d4f19dc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5177fe640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5177fe640 1 -- 172.21.15.64:0/3810568528 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53003b070 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5177fe640 1 -- 172.21.15.64:0/3810568528 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe530028610 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5280b1040 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5177fe640 1 -- 172.21.15.64:0/3810568528 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe530024070 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5408c2740 10 client.25031 fetch_fsmap learned FSMap version 27 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a8682fede0 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.563+0000 7fe5177fe640 1 -- 172.21.15.64:0/3810568528 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5280b1040 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5177fe640 1 -- 172.21.15.64:0/3810568528 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe530004c00 con 0x55a868284310 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5408c2740 10 client.25031 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:35:36.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5408c2740 10 client.25031 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:35:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5408c2740 20 client.25031 populate_metadata read hostname 'smithi064' 2024-08-19T01:35:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5408c2740 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868301520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5408c2740 4 client.25031 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-08-19T01:35:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x55a8681ec1a0 con 0x55a8682940d0 2024-08-19T01:35:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe537fff640 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868301520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe5177fe640 10 client.25031 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.564+0000 7fe537fff640 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868301520 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53781dec1809c66f server_cookie=2a6cab45df0f030d in_seq=0 out_seq=0 2024-08-19T01:35:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5177fe640 1 -- 172.21.15.64:0/3810568528 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55a8681ec1a0 con 0x55a8682940d0 2024-08-19T01:35:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5177fe640 10 client.25031 handle_command_reply: tid=0 2024-08-19T01:35:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 client.25031 shutdown 2024-08-19T01:35:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868284310 msgr2=0x55a8682882c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868284310 0x55a8682882c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe530009bb0 tx=0x7fe530028d90 comp rx=0 tx=0).stop 2024-08-19T01:35:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 shutdown_connections 2024-08-19T01:35:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a8682940d0 0x55a868301520 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a868284310 0x55a8682882c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a868279a50 0x55a868287d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5177fe640 0 client.25031 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 --2- 172.21.15.64:0/3810568528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a868279130 0x55a86828ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 >> 172.21.15.64:0/3810568528 conn(0x55a8682756c0 msgr2=0x55a8682746f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 shutdown_connections 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 1 -- 172.21.15.64:0/3810568528 wait complete. 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.650+0000 7fe5408c2740 20 client.25031 trim_cache size 0 max 16384 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:35:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 27, 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 6, 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "971a494f": { 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 6, 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:35:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 130.0, 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:36.655 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5200a2cf0 msgr2=0x7fe5200a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5200a2cf0 0x7fe5200a51b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe530005fd0 tx=0x7fe530009e90 comp rx=0 tx=0).stop 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538102ac0 msgr2=0x7fe53819a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538102ac0 0x7fe53819a500 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ea60 tx=0x7fe52800ef20 comp rx=0 tx=0).stop 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 shutdown_connections 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe5200a2cf0 0x7fe5200a51b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe538107fc0 0x7fe53819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538103470 0x7fe53819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 --2- 172.21.15.64:0/1567295798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe538102ac0 0x7fe53819a500 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:36.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 >> 172.21.15.64:0/1567295798 conn(0x7fe5380fc500 msgr2=0x7fe538100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:36.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 shutdown_connections 2024-08-19T01:35:36.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:36.651+0000 7fe53e4c4640 1 -- 172.21.15.64:0/1567295798 wait complete. 2024-08-19T01:35:37.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 15.35 MB/sec warmup 69 sec latency 725.467 ms 2024-08-19T01:35:37.732 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:37 smithi064 ceph-mon[18515]: pgmap v74: 129 pgs: 129 active+clean; 109 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.4 MiB/s wr, 311 op/s 2024-08-19T01:35:37.767 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '971a494f', quiesce took 4.4 seconds. Will release after: 5.6 2024-08-19T01:35:37.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:37 smithi123 ceph-mon[21829]: pgmap v74: 129 pgs: 129 active+clean; 109 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.4 MiB/s wr, 311 op/s 2024-08-19T01:35:37.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:37 smithi161 ceph-mon[21757]: pgmap v74: 129 pgs: 129 active+clean; 109 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.4 MiB/s wr, 311 op/s 2024-08-19T01:35:38.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 15.13 MB/sec warmup 70 sec latency 1725.549 ms 2024-08-19T01:35:39.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 14.91 MB/sec warmup 71 sec latency 2725.674 ms 2024-08-19T01:35:39.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:39 smithi123 ceph-mon[21829]: pgmap v75: 129 pgs: 129 active+clean; 109 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-08-19T01:35:39.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:39 smithi161 ceph-mon[21757]: pgmap v75: 129 pgs: 129 active+clean; 109 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-08-19T01:35:39.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:39 smithi064 ceph-mon[18515]: pgmap v75: 129 pgs: 129 active+clean; 109 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-08-19T01:35:40.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 14.71 MB/sec warmup 72 sec latency 3725.801 ms 2024-08-19T01:35:41.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 14.51 MB/sec warmup 73 sec latency 4725.912 ms 2024-08-19T01:35:41.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:41 smithi123 ceph-mon[21829]: pgmap v76: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 281 op/s 2024-08-19T01:35:41.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:41 smithi161 ceph-mon[21757]: pgmap v76: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 281 op/s 2024-08-19T01:35:41.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:41 smithi064 ceph-mon[18515]: pgmap v76: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 281 op/s 2024-08-19T01:35:42.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 14.31 MB/sec warmup 74 sec latency 5726.030 ms 2024-08-19T01:35:43.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:35:43.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 14.12 MB/sec warmup 75 sec latency 6726.065 ms 2024-08-19T01:35:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:35:43.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:43 smithi064 ceph-mon[18515]: pgmap v77: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 186 op/s 2024-08-19T01:35:43.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:43 smithi123 ceph-mon[21829]: pgmap v77: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 186 op/s 2024-08-19T01:35:43.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:43 smithi161 ceph-mon[21757]: pgmap v77: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 186 op/s 2024-08-19T01:35:44.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 -- 172.21.15.64:0/3971112782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 msgr2=0x7f799810d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:44.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 --2- 172.21.15.64:0/3971112782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799810d770 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7990009ca0 tx=0x7f799002a950 comp rx=0 tx=0).stop 2024-08-19T01:35:44.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 -- 172.21.15.64:0/3971112782 shutdown_connections 2024-08-19T01:35:44.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 --2- 172.21.15.64:0/3971112782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799810d770 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 --2- 172.21.15.64:0/3971112782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79981022a0 0x7f79981026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 --2- 172.21.15.64:0/3971112782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998100b50 0x7f7998100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 -- 172.21.15.64:0/3971112782 >> 172.21.15.64:0/3971112782 conn(0x7f79980fc500 msgr2=0x7f79980fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 -- 172.21.15.64:0/3971112782 shutdown_connections 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.102+0000 7f799efdf640 1 -- 172.21.15.64:0/3971112782 wait complete. 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 Processor -- start 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 -- start start 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7998100b50 0x7f7998197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981022a0 0x7f79981975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799819b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7998110cb0 con 0x7f7998102c20 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7998110b30 con 0x7f7998100b50 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799efdf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7998110e30 con 0x7f79981022a0 2024-08-19T01:35:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799d57c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799819b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799d57c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799819b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51832/0 (socket says 172.21.15.64:51832) 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799d57c640 1 -- 172.21.15.64:0/1678999165 learned_addr learned my addr 172.21.15.64:0/1678999165 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.103+0000 7f799cd7b640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7998100b50 0x7f7998197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f7997fff640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981022a0 0x7f79981975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799d57c640 1 -- 172.21.15.64:0/1678999165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981022a0 msgr2=0x7f79981975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799d57c640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981022a0 0x7f79981975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799d57c640 1 -- 172.21.15.64:0/1678999165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7998100b50 msgr2=0x7f7998197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799d57c640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7998100b50 0x7f7998197070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799d57c640 1 -- 172.21.15.64:0/1678999165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799819c020 con 0x7f7998102c20 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799d57c640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799819b940 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7990004af0 tx=0x7f799002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f7995ffb640 1 -- 172.21.15.64:0/1678999165 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990034040 con 0x7f7998102c20 2024-08-19T01:35:44.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799819c2b0 con 0x7f7998102c20 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f7995ffb640 1 -- 172.21.15.64:0/1678999165 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7990030920 con 0x7f7998102c20 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f7995ffb640 1 -- 172.21.15.64:0/1678999165 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990004820 con 0x7f7998102c20 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.104+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799819c730 con 0x7f7998102c20 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.105+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7998100f30 con 0x7f7998102c20 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.107+0000 7f7995ffb640 1 -- 172.21.15.64:0/1678999165 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7990037020 con 0x7f7998102c20 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.107+0000 7f7995ffb640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f79880a2cf0 0x7f79880a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.108+0000 7f799cd7b640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f79880a2cf0 0x7f79880a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.108+0000 7f7995ffb640 1 -- 172.21.15.64:0/1678999165 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f79900f1f90 con 0x7f7998102c20 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.108+0000 7f799cd7b640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f79880a2cf0 0x7f79880a51b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f798c005d00 tx=0x7f798c00b410 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:44.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.112+0000 7f7995ffb640 1 -- 172.21.15.64:0/1678999165 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f7990040070 con 0x7f7998102c20 2024-08-19T01:35:44.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.321+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7998069f00 con 0x7f7998102c20 2024-08-19T01:35:44.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.322+0000 7f7995ffb640 1 -- 172.21.15.64:0/1678999165 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11924 (secure 0 0 0) 0x7f79900be920 con 0x7f7998102c20 2024-08-19T01:35:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:35:44.323 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"btime":"2024-08-19T01:34:36:714408+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:34:36.547976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:35:44.324 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2024-08-19T01:35:44.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f79880a2cf0 msgr2=0x7f79880a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:44.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f79880a2cf0 0x7f79880a51b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f798c005d00 tx=0x7f798c00b410 comp rx=0 tx=0).stop 2024-08-19T01:35:44.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 msgr2=0x7f799819b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:44.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799819b940 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7990004af0 tx=0x7f799002f5b0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 shutdown_connections 2024-08-19T01:35:44.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f79880a2cf0 0x7f79880a51b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7998102c20 0x7f799819b940 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981022a0 0x7f79981975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 --2- 172.21.15.64:0/1678999165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7998100b50 0x7f7998197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:44.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 >> 172.21.15.64:0/1678999165 conn(0x7f79980fc500 msgr2=0x7f7998106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:44.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 shutdown_connections 2024-08-19T01:35:44.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:44.324+0000 7f799efdf640 1 -- 172.21.15.64:0/1678999165 wait complete. 2024-08-19T01:35:44.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 13.93 MB/sec warmup 76 sec latency 7726.159 ms 2024-08-19T01:35:44.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:44 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1678999165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:35:44.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:44 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1678999165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:35:44.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:44 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1678999165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:35:45.079 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 971a494f --release --await' 2024-08-19T01:35:45.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 971a494f --release --await 2024-08-19T01:35:45.376 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:35:45.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 193358 13.75 MB/sec warmup 77 sec latency 8726.205 ms 2024-08-19T01:35:45.788 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:45 smithi064 ceph-mon[18515]: pgmap v78: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 186 op/s 2024-08-19T01:35:45.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 -- 172.21.15.64:0/3626226500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1018b0 msgr2=0x7f64ec101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3626226500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1018b0 0x7f64ec101d10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009930 tx=0x7f64e002a420 comp rx=0 tx=0).stop 2024-08-19T01:35:45.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 -- 172.21.15.64:0/3626226500 shutdown_connections 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3626226500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 0x7f64ec10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3626226500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1018b0 0x7f64ec101d10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3626226500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64ec1078b0 0x7f64ec107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 -- 172.21.15.64:0/3626226500 >> 172.21.15.64:0/3626226500 conn(0x7f64ec0fd5f0 msgr2=0x7f64ec0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 -- 172.21.15.64:0/3626226500 shutdown_connections 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.874+0000 7f64f27b9640 1 -- 172.21.15.64:0/3626226500 wait complete. 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 Processor -- start 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 -- start start 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64ec1018b0 0x7f64ec198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 0x7f64ec1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1078b0 0x7f64ec19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec111cb0 con 0x7f64ec1078b0 2024-08-19T01:35:45.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec111b30 con 0x7f64ec1018b0 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.875+0000 7f64f27b9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec111e30 con 0x7f64ec102250 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64f0d56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1078b0 0x7f64ec19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 0x7f64ec1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64f0d56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1078b0 0x7f64ec19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51848/0 (socket says 172.21.15.64:51848) 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64f0d56640 1 -- 172.21.15.64:0/3329861997 learned_addr learned my addr 172.21.15.64:0/3329861997 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 0x7f64ec1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:45096/0 (socket says 172.21.15.64:45096) 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64ebfff640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64ec1018b0 0x7f64ec198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 -- 172.21.15.64:0/3329861997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64ec1018b0 msgr2=0x7f64ec198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64ec1018b0 0x7f64ec198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 -- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1078b0 msgr2=0x7f64ec19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1078b0 0x7f64ec19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 -- 172.21.15.64:0/3329861997 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec19d130 con 0x7f64ec102250 2024-08-19T01:35:45.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 0x7f64ec1986c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f64e0005940 tx=0x7f64e00040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64e97fa640 1 -- 172.21.15.64:0/3329861997 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e003f070 con 0x7f64ec102250 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64e97fa640 1 -- 172.21.15.64:0/3329861997 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e00044c0 con 0x7f64ec102250 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec19d3c0 con 0x7f64ec102250 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64e97fa640 1 -- 172.21.15.64:0/3329861997 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0035070 con 0x7f64ec102250 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.876+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec1a4d00 con 0x7f64ec102250 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 Processor -- start 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64e97fa640 1 -- 172.21.15.64:0/3329861997 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f64e003b460 con 0x7f64ec102250 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 -- start start 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f216180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f220a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f220f80 0x55b00f223370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b00f177c60 con 0x55b00f220f80 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55b00f10fb90 con 0x55b00f2166c0 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b00f173c20 con 0x55b00f215da0 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f220f80 0x55b00f223370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f220a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f220a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:48254/0 (socket says 172.21.15.64:48254) 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64ebfff640 1 -- 172.21.15.64:0/3421060645 learned_addr learned my addr 172.21.15.64:0/3421060645 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f220f80 0x55b00f223370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51854/0 (socket says 172.21.15.64:51854) 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64f0d56640 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f216180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.879+0000 7f64e97fa640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f64d40a2cf0 0x7f64d40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64ebfff640 1 -- 172.21.15.64:0/3421060645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 msgr2=0x55b00f216180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64ebfff640 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f216180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64ebfff640 1 -- 172.21.15.64:0/3421060645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f220f80 msgr2=0x55b00f223370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64ebfff640 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f220f80 0x55b00f223370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64ebfff640 1 -- 172.21.15.64:0/3421060645 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b00ee87580 con 0x55b00f2166c0 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64f0d56640 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f216180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64f0d56640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f64d40a2cf0 0x7f64d40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64e97fa640 1 -- 172.21.15.64:0/3329861997 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f64e00efd00 con 0x7f64ec102250 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64ebfff640 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f220a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00bc60 tx=0x7f64dc00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73178628ce758127 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64d27fc640 1 -- 172.21.15.64:0/3421060645 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc015070 con 0x55b00f2166c0 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64f0d56640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f64d40a2cf0 0x7f64d40a51b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64e40049d0 tx=0x7f64e4004a40 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64d27fc640 1 -- 172.21.15.64:0/3421060645 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64dc011920 con 0x55b00f2166c0 2024-08-19T01:35:45.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.880+0000 7f64d27fc640 1 -- 172.21.15.64:0/3421060645 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc010830 con 0x55b00f2166c0 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3421060645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 msgr2=0x55b00f220a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f220a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00bc60 tx=0x7f64dc00f730 comp rx=0 tx=0).stop 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3421060645 shutdown_connections 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f220f80 0x55b00f223370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f220a40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3421060645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f216180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3421060645 >> 172.21.15.64:0/3421060645 conn(0x55b00f212330 msgr2=0x55b00f212710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3421060645 shutdown_connections 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3421060645 wait complete. 2024-08-19T01:35:45.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 -- start start 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.881+0000 7f64f4bb7740 1 Processor -- start 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f21ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f21e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f21e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b00f223b50 con 0x55b00f2166c0 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55b00f1157c0 con 0x55b00f220f80 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b00f223cd0 con 0x55b00f215da0 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f4bb7740 10 client.0 _ll_register_callbacks cb 0x55b00f11ef50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f21e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f21e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51870/0 (socket says 172.21.15.64:51870) 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 -- 172.21.15.64:0/1818294425 learned_addr learned my addr 172.21.15.64:0/1818294425 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64f0d56640 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f21e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64ebfff640 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f21ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 -- 172.21.15.64:0/1818294425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 msgr2=0x55b00f21ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f21ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 -- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 msgr2=0x55b00f21e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f21e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 -- 172.21.15.64:0/1818294425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b00ef51960 con 0x55b00f2166c0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f21e330 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f64e00ed6a0 tx=0x7f64e00b7ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb238d042973927c server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.882+0000 7f64cb7fe640 1 -- 172.21.15.64:0/1818294425 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0048040 con 0x55b00f2166c0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64cb7fe640 1 -- 172.21.15.64:0/1818294425 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f64e00c4650 con 0x55b00f2166c0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64dc011920 con 0x55b00f2166c0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64cb7fe640 1 -- 172.21.15.64:0/1818294425 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f64e00bb7d0 con 0x55b00f2166c0 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 10 client.15561 fetch_fsmap learned FSMap version 27 2024-08-19T01:35:45.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b00f22fb10 con 0x55b00f2166c0 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64cb7fe640 1 -- 172.21.15.64:0/1818294425 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64dc011920 con 0x55b00f2166c0 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64cb7fe640 1 -- 172.21.15.64:0/1818294425 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f64e00f3020 con 0x55b00f2166c0 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 10 client.15561 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 10 client.15561 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 20 client.15561 populate_metadata read hostname 'smithi064' 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f231020 0x55b00f233460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 4 client.15561 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b00f1851d0 con 0x55b00f231020 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.883+0000 7f64ebfff640 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f231020 0x55b00f233460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.884+0000 7f64ebfff640 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f231020 0x55b00f233460 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad2540881cccfe1b server_cookie=e9004443fd0648cb in_seq=0 out_seq=0 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.884+0000 7f64cb7fe640 10 client.15561 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.884+0000 7f64cb7fe640 1 -- 172.21.15.64:0/1818294425 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b00f1851d0 con 0x55b00f231020 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.884+0000 7f64cb7fe640 10 client.15561 handle_command_reply: tid=0 2024-08-19T01:35:45.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.884+0000 7f64f4bb7740 1 client.15561 shutdown 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 msgr2=0x55b00f21e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f21e330 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f64e00ed6a0 tx=0x7f64e00b7ac0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 shutdown_connections 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f231020 0x55b00f233460 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f21e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f21e330 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64cb7fe640 0 client.15561 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/1818294425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f21ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 >> 172.21.15.64:0/1818294425 conn(0x55b00f212330 msgr2=0x55b00f20d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 shutdown_connections 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 1 -- 172.21.15.64:0/1818294425 wait complete. 2024-08-19T01:35:45.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.885+0000 7f64f4bb7740 20 client.15561 trim_cache size 0 max 16384 2024-08-19T01:35:45.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 Processor -- start 2024-08-19T01:35:45.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 -- start start 2024-08-19T01:35:45.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f215da0 0x55b00f238d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f239260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f220f80 0x55b00f21b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1af360 con 0x55b00f215da0 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1acc30 con 0x55b00f2166c0 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1aaa80 con 0x55b00f220f80 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f215da0 0x55b00f238d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f215da0 0x55b00f238d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51874/0 (socket says 172.21.15.64:51874) 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 -- 172.21.15.64:0/3394616987 learned_addr learned my addr 172.21.15.64:0/3394616987 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f239260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 -- 172.21.15.64:0/3394616987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f220f80 msgr2=0x55b00f21b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f220f80 0x55b00f21b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 -- 172.21.15.64:0/3394616987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 msgr2=0x55b00f239260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f239260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64ebfff640 1 -- 172.21.15.64:0/3394616987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b00f212fc0 con 0x55b00f215da0 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.902+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f239260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64ebfff640 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f215da0 0x55b00f238d20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00bc60 tx=0x7f64dc0103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5501d06e002fe797 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64d1ffb640 1 -- 172.21.15.64:0/3394616987 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc015040 con 0x55b00f215da0 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64d1ffb640 1 -- 172.21.15.64:0/3394616987 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64dc00b3a0 con 0x55b00f215da0 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64d1ffb640 1 -- 172.21.15.64:0/3394616987 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc004d90 con 0x55b00f215da0 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3394616987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f215da0 msgr2=0x55b00f238d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f215da0 0x55b00f238d20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00bc60 tx=0x7f64dc0103d0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3394616987 shutdown_connections 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f220f80 0x55b00f21b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f2166c0 0x55b00f239260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/3394616987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f215da0 0x55b00f238d20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.903+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3394616987 >> 172.21.15.64:0/3394616987 conn(0x55b00f212330 msgr2=0x55b00f21fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:45.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3394616987 shutdown_connections 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 -- 172.21.15.64:0/3394616987 wait complete. 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 -- start start 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 Processor -- start 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f228f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f229470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f2253e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1ab0f0 con 0x55b00f2166c0 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1ac560 con 0x55b00f220f80 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.904+0000 7f64f4bb7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1ab7c0 con 0x55b00f215da0 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64f4bb7740 10 client.0 _ll_register_callbacks cb 0x7f64e0005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f229470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f229470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51876/0 (socket says 172.21.15.64:51876) 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 -- 172.21.15.64:0/313587044 learned_addr learned my addr 172.21.15.64:0/313587044 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:35:45.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64ebfff640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f228f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64f0d56640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f2253e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 -- 172.21.15.64:0/313587044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 msgr2=0x55b00f228f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f228f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 -- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 msgr2=0x55b00f2253e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f2253e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 -- 172.21.15.64:0/313587044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b00f21edc0 con 0x55b00f2166c0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64ebfff640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f228f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64eb7fe640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f229470 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f64e00c4a30 tx=0x7f64e00ed150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4595a4d6b09cb77a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64cb7fe640 1 -- 172.21.15.64:0/313587044 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0048070 con 0x55b00f2166c0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64cb7fe640 1 -- 172.21.15.64:0/313587044 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f64e0031dd0 con 0x55b00f2166c0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64dc00b3a0 con 0x55b00f2166c0 2024-08-19T01:35:45.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64cb7fe640 1 -- 172.21.15.64:0/313587044 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f64e00f3330 con 0x55b00f2166c0 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64f4bb7740 10 client.15567 fetch_fsmap learned FSMap version 27 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.905+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b00f218c10 con 0x55b00f2166c0 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64cb7fe640 1 -- 172.21.15.64:0/313587044 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64dc00b3a0 con 0x55b00f2166c0 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64cb7fe640 1 -- 172.21.15.64:0/313587044 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f64e004b060 con 0x55b00f2166c0 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64f4bb7740 10 client.15567 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64f4bb7740 10 client.15567 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64f4bb7740 20 client.15567 populate_metadata read hostname 'smithi064' 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f29c750 0x55b00f29eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64f4bb7740 4 client.15567 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "971a494f", "release": true, "await": true}] 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "971a494f", "release": true, "await": true}) -- 0x55b00f17f6e0 con 0x55b00f29c750 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64ebfff640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f29c750 0x55b00f29eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64cb7fe640 10 client.15567 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:45.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.906+0000 7f64ebfff640 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f29c750 0x55b00f29eb70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9a5e4262db6c450 server_cookie=57275c3d55958246 in_seq=0 out_seq=0 2024-08-19T01:35:45.916 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:45 smithi123 ceph-mon[21829]: pgmap v78: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 186 op/s 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64cb7fe640 1 -- 172.21.15.64:0/313587044 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55b00f17f6e0 con 0x55b00f29c750 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64cb7fe640 10 client.15567 handle_command_reply: tid=0 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 client.15567 shutdown 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 msgr2=0x55b00f229470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f229470 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f64e00c4a30 tx=0x7f64e00ed150 comp rx=0 tx=0).stop 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 shutdown_connections 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55b00f29c750 0x55b00f29eb70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64cb7fe640 0 client.15567 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:35:45.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55b00f220f80 0x55b00f2253e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b00f2166c0 0x55b00f229470 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 --2- 172.21.15.64:0/313587044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b00f215da0 0x55b00f228f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.917+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 >> 172.21.15.64:0/313587044 conn(0x55b00f212330 msgr2=0x55b00f20c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 shutdown_connections 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f4bb7740 1 -- 172.21.15.64:0/313587044 wait complete. 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f4bb7740 20 client.15567 trim_cache size 0 max 16384 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 27, 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:35:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 8, 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "971a494f": { 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 8, 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 130.0, 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:35:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:35:45.923 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f64d40a2cf0 msgr2=0x7f64d40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f64d40a2cf0 0x7f64d40a51b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64e40049d0 tx=0x7f64e4004a40 comp rx=0 tx=0).stop 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 msgr2=0x7f64ec1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 0x7f64ec1986c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f64e0005940 tx=0x7f64e00040c0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 shutdown_connections 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f64d40a2cf0 0x7f64d40a51b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ec1078b0 0x7f64ec19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64ec102250 0x7f64ec1986c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 --2- 172.21.15.64:0/3329861997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64ec1018b0 0x7f64ec198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:35:45.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 >> 172.21.15.64:0/3329861997 conn(0x7f64ec0fd5f0 msgr2=0x7f64ec0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:35:45.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 shutdown_connections 2024-08-19T01:35:45.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:35:45.918+0000 7f64f27b9640 1 -- 172.21.15.64:0/3329861997 wait complete. 2024-08-19T01:35:45.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:45 smithi161 ceph-mon[21757]: pgmap v78: 129 pgs: 129 active+clean; 110 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 186 op/s 2024-08-19T01:35:46.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 195055 13.73 MB/sec warmup 78 sec latency 9270.283 ms 2024-08-19T01:35:46.895 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '971a494f', total seconds elapsed: 13.6 2024-08-19T01:35:47.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 197564 13.68 MB/sec warmup 79 sec latency 10.392 ms 2024-08-19T01:35:47.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:47 smithi123 ceph-mon[21829]: pgmap v79: 129 pgs: 129 active+clean; 108 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-08-19T01:35:47.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:47 smithi161 ceph-mon[21757]: pgmap v79: 129 pgs: 129 active+clean; 108 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-08-19T01:35:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:47 smithi064 ceph-mon[18515]: pgmap v79: 129 pgs: 129 active+clean; 108 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-08-19T01:35:48.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 199871 13.67 MB/sec warmup 80 sec latency 17.341 ms 2024-08-19T01:35:49.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 202862 13.72 MB/sec warmup 81 sec latency 12.145 ms 2024-08-19T01:35:49.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:49 smithi123 ceph-mon[21829]: pgmap v80: 129 pgs: 129 active+clean; 108 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-08-19T01:35:49.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:49 smithi161 ceph-mon[21757]: pgmap v80: 129 pgs: 129 active+clean; 108 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-08-19T01:35:49.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:49 smithi064 ceph-mon[18515]: pgmap v80: 129 pgs: 129 active+clean; 108 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-08-19T01:35:50.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 206041 13.78 MB/sec warmup 82 sec latency 7.377 ms 2024-08-19T01:35:51.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 208129 13.70 MB/sec warmup 83 sec latency 63.999 ms 2024-08-19T01:35:51.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:51 smithi123 ceph-mon[21829]: pgmap v81: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-08-19T01:35:51.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:51 smithi161 ceph-mon[21757]: pgmap v81: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-08-19T01:35:51.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:51 smithi064 ceph-mon[18515]: pgmap v81: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-08-19T01:35:52.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 210766 13.74 MB/sec warmup 84 sec latency 12.915 ms 2024-08-19T01:35:53.377 INFO:tasks.workunit.client.0.smithi064.stdout: 1 213599 13.75 MB/sec warmup 85 sec latency 15.445 ms 2024-08-19T01:35:53.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:53 smithi123 ceph-mon[21829]: pgmap v82: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-08-19T01:35:53.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:53 smithi161 ceph-mon[21757]: pgmap v82: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-08-19T01:35:53.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:53 smithi064 ceph-mon[18515]: pgmap v82: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-08-19T01:35:54.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 216823 13.81 MB/sec warmup 86 sec latency 12.791 ms 2024-08-19T01:35:55.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 219949 13.85 MB/sec warmup 87 sec latency 17.122 ms 2024-08-19T01:35:55.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:55 smithi123 ceph-mon[21829]: pgmap v83: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-08-19T01:35:55.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:55 smithi161 ceph-mon[21757]: pgmap v83: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-08-19T01:35:55.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:55 smithi064 ceph-mon[18515]: pgmap v83: 129 pgs: 129 active+clean; 113 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-08-19T01:35:56.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 221471 13.77 MB/sec warmup 88 sec latency 88.283 ms 2024-08-19T01:35:57.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 224437 13.78 MB/sec warmup 89 sec latency 11.642 ms 2024-08-19T01:35:57.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:57 smithi123 ceph-mon[21829]: pgmap v84: 129 pgs: 129 active+clean; 119 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 269 op/s 2024-08-19T01:35:57.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:57 smithi161 ceph-mon[21757]: pgmap v84: 129 pgs: 129 active+clean; 119 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 269 op/s 2024-08-19T01:35:57.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:57 smithi064 ceph-mon[18515]: pgmap v84: 129 pgs: 129 active+clean; 119 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 269 op/s 2024-08-19T01:35:58.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 227513 13.83 MB/sec warmup 90 sec latency 8.942 ms 2024-08-19T01:35:59.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 230811 13.89 MB/sec warmup 91 sec latency 14.309 ms 2024-08-19T01:35:59.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:35:59 smithi123 ceph-mon[21829]: pgmap v85: 129 pgs: 129 active+clean; 119 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 231 op/s 2024-08-19T01:35:59.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:35:59 smithi161 ceph-mon[21757]: pgmap v85: 129 pgs: 129 active+clean; 119 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 231 op/s 2024-08-19T01:35:59.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:35:59 smithi064 ceph-mon[18515]: pgmap v85: 129 pgs: 129 active+clean; 119 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 231 op/s 2024-08-19T01:36:00.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 233780 13.88 MB/sec warmup 92 sec latency 20.155 ms 2024-08-19T01:36:01.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 235283 13.82 MB/sec warmup 93 sec latency 101.799 ms 2024-08-19T01:36:01.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:01 smithi123 ceph-mon[21829]: pgmap v86: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 359 op/s 2024-08-19T01:36:01.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:01 smithi161 ceph-mon[21757]: pgmap v86: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 359 op/s 2024-08-19T01:36:01.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:01 smithi064 ceph-mon[18515]: pgmap v86: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 359 op/s 2024-08-19T01:36:02.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 238254 13.85 MB/sec warmup 94 sec latency 15.870 ms 2024-08-19T01:36:03.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 241505 13.91 MB/sec warmup 95 sec latency 18.497 ms 2024-08-19T01:36:03.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:03 smithi123 ceph-mon[21829]: pgmap v87: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-08-19T01:36:03.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:03 smithi161 ceph-mon[21757]: pgmap v87: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-08-19T01:36:03.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:03 smithi064 ceph-mon[18515]: pgmap v87: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-08-19T01:36:04.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 244588 13.91 MB/sec warmup 96 sec latency 20.900 ms 2024-08-19T01:36:05.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 247184 13.89 MB/sec warmup 97 sec latency 18.297 ms 2024-08-19T01:36:05.467 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-19T01:36:05.467 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-08-19T01:36:05.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:36:05.747 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:36:05.870 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:05 smithi064 ceph-mon[18515]: pgmap v88: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-08-19T01:36:05.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:05 smithi123 ceph-mon[21829]: pgmap v88: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-08-19T01:36:05.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:05 smithi161 ceph-mon[21757]: pgmap v88: 129 pgs: 129 active+clean; 125 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 251 op/s 2024-08-19T01:36:06.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 249027 13.88 MB/sec warmup 98 sec latency 63.642 ms 2024-08-19T01:36:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 -- 172.21.15.64:0/2319359126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660103410 msgr2=0x7f46601078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 --2- 172.21.15.64:0/2319359126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660103410 0x7f46601078c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4648009930 tx=0x7f464802a420 comp rx=0 tx=0).stop 2024-08-19T01:36:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 -- 172.21.15.64:0/2319359126 shutdown_connections 2024-08-19T01:36:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 --2- 172.21.15.64:0/2319359126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4660107fd0 0x7f466010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 --2- 172.21.15.64:0/2319359126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660103410 0x7f46601078c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 --2- 172.21.15.64:0/2319359126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4660102a60 0x7f4660102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 -- 172.21.15.64:0/2319359126 >> 172.21.15.64:0/2319359126 conn(0x7f46600fc500 msgr2=0x7f46600fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 -- 172.21.15.64:0/2319359126 shutdown_connections 2024-08-19T01:36:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.388+0000 7f4664f83640 1 -- 172.21.15.64:0/2319359126 wait complete. 2024-08-19T01:36:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 Processor -- start 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 -- start start 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4660102a60 0x7f46601982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4660103410 0x7f4660198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660107fd0 0x7f466019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4660111dd0 con 0x7f4660107fd0 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4660111c50 con 0x7f4660102a60 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4664f83640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4660111f50 con 0x7f4660103410 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4657fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4660103410 0x7f4660198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f465e59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4660102a60 0x7f46601982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4657fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4660103410 0x7f4660198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:51024/0 (socket says 172.21.15.64:51024) 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f4657fff640 1 -- 172.21.15.64:0/2953997179 learned_addr learned my addr 172.21.15.64:0/2953997179 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f465ed9d640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660107fd0 0x7f466019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f465e59c640 1 -- 172.21.15.64:0/2953997179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4660103410 msgr2=0x7f4660198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f465e59c640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4660103410 0x7f4660198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f465e59c640 1 -- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660107fd0 msgr2=0x7f466019cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f465e59c640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660107fd0 0x7f466019cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.389+0000 7f465e59c640 1 -- 172.21.15.64:0/2953997179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466019d2a0 con 0x7f4660102a60 2024-08-19T01:36:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.390+0000 7f465e59c640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4660102a60 0x7f46601982f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4650009bc0 tx=0x7f465000b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.390+0000 7f46577fe640 1 -- 172.21.15.64:0/2953997179 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465001e070 con 0x7f4660102a60 2024-08-19T01:36:06.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.390+0000 7f46577fe640 1 -- 172.21.15.64:0/2953997179 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46500198f0 con 0x7f4660102a60 2024-08-19T01:36:06.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.390+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466019d590 con 0x7f4660102a60 2024-08-19T01:36:06.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.390+0000 7f46577fe640 1 -- 172.21.15.64:0/2953997179 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46500187e0 con 0x7f4660102a60 2024-08-19T01:36:06.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.390+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46601005f0 con 0x7f4660102a60 2024-08-19T01:36:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.391+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f466006a580 con 0x7f4660102a60 2024-08-19T01:36:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.392+0000 7f46577fe640 1 -- 172.21.15.64:0/2953997179 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4650021070 con 0x7f4660102a60 2024-08-19T01:36:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.393+0000 7f46577fe640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f464c0a2cf0 0x7f464c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.393+0000 7f46577fe640 1 -- 172.21.15.64:0/2953997179 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f46500d3310 con 0x7f4660102a60 2024-08-19T01:36:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.393+0000 7f4657fff640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f464c0a2cf0 0x7f464c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.394+0000 7f4657fff640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f464c0a2cf0 0x7f464c0a51b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4648000c00 tx=0x7f46480058d0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:06.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.396+0000 7f46577fe640 1 -- 172.21.15.64:0/2953997179 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f465009f530 con 0x7f4660102a60 2024-08-19T01:36:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.633+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4660102e40 con 0x7f4660102a60 2024-08-19T01:36:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.634+0000 7f46577fe640 1 -- 172.21.15.64:0/2953997179 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11924 (secure 0 0 0) 0x7f46500a3830 con 0x7f4660102a60 2024-08-19T01:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"btime":"2024-08-19T01:34:36:714408+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:34:36.547976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:36:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f464c0a2cf0 msgr2=0x7f464c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f464c0a2cf0 0x7f464c0a51b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4648000c00 tx=0x7f46480058d0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4660102a60 msgr2=0x7f46601982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4660102a60 0x7f46601982f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4650009bc0 tx=0x7f465000b690 comp rx=0 tx=0).stop 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 shutdown_connections 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f464c0a2cf0 0x7f464c0a51b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4660107fd0 0x7f466019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4660103410 0x7f4660198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 --2- 172.21.15.64:0/2953997179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4660102a60 0x7f46601982f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 >> 172.21.15.64:0/2953997179 conn(0x7f46600fc500 msgr2=0x7f4660105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 shutdown_connections 2024-08-19T01:36:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:06.636+0000 7f4664f83640 1 -- 172.21.15.64:0/2953997179 wait complete. 2024-08-19T01:36:06.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:06 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2953997179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:06.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:06 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2953997179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:06.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:06 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2953997179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:07.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 252102 13.92 MB/sec warmup 99 sec latency 18.113 ms 2024-08-19T01:36:07.384 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-08-19T01:36:07.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-08-19T01:36:07.656 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:36:07.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:07 smithi123 ceph-mon[21829]: pgmap v89: 129 pgs: 129 active+clean; 131 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.4 MiB/s wr, 389 op/s 2024-08-19T01:36:07.929 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:07 smithi064 ceph-mon[18515]: pgmap v89: 129 pgs: 129 active+clean; 131 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.4 MiB/s wr, 389 op/s 2024-08-19T01:36:07.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:07 smithi161 ceph-mon[21757]: pgmap v89: 129 pgs: 129 active+clean; 131 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.4 MiB/s wr, 389 op/s 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.204+0000 7fedec2c4640 1 -- 172.21.15.64:0/2297390090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 msgr2=0x7fede410e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.204+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2297390090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede410e9a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7feddc009cd0 tx=0x7feddc02a8d0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 -- 172.21.15.64:0/2297390090 shutdown_connections 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2297390090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede410e9a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2297390090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fede41034b0 0x7fede41078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2297390090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede4102b00 0x7fede4102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 -- 172.21.15.64:0/2297390090 >> 172.21.15.64:0/2297390090 conn(0x7fede40fc500 msgr2=0x7fede40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:08.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 -- 172.21.15.64:0/2297390090 shutdown_connections 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 -- 172.21.15.64:0/2297390090 wait complete. 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 Processor -- start 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.205+0000 7fedec2c4640 1 -- start start 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedec2c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fede4102b00 0x7fede4198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedec2c4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede41034b0 0x7fede4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedec2c4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede419cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedec2c4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fede4111e00 con 0x7fede4108030 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedec2c4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fede4111c80 con 0x7fede4102b00 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedec2c4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fede4111f80 con 0x7fede41034b0 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede419cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede419cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44758/0 (socket says 172.21.15.64:44758) 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea060640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fede4102b00 0x7fede4198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fede985f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede41034b0 0x7fede4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 -- 172.21.15.64:0/2351199358 learned_addr learned my addr 172.21.15.64:0/2351199358 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:08.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 -- 172.21.15.64:0/2351199358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede41034b0 msgr2=0x7fede4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede41034b0 0x7fede4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 -- 172.21.15.64:0/2351199358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fede4102b00 msgr2=0x7fede4198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fede4102b00 0x7fede4198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 -- 172.21.15.64:0/2351199358 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede419d2d0 con 0x7fede4108030 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fede985f640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede41034b0 0x7fede4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.206+0000 7fedea861640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede419cbf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7feddc009ca0 tx=0x7feddc02f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.207+0000 7feddb7fe640 1 -- 172.21.15.64:0/2351199358 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc031760 con 0x7fede4108030 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.207+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede419d560 con 0x7fede4108030 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.207+0000 7feddb7fe640 1 -- 172.21.15.64:0/2351199358 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feddc003e50 con 0x7fede4108030 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.207+0000 7feddb7fe640 1 -- 172.21.15.64:0/2351199358 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc030590 con 0x7fede4108030 2024-08-19T01:36:08.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.207+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede41a4ea0 con 0x7fede4108030 2024-08-19T01:36:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.209+0000 7feddb7fe640 1 -- 172.21.15.64:0/2351199358 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feddc004040 con 0x7fede4108030 2024-08-19T01:36:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.209+0000 7fedee6c2740 1 Processor -- start 2024-08-19T01:36:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedee6c2740 1 -- start start 2024-08-19T01:36:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x561136352510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedee6c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5611362b7a70 con 0x561136352a50 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedee6c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56113624f780 con 0x561136352130 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedee6c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5611362b3a30 con 0x56113635d310 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7feddb7fe640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fedd00a2cf0 0x7fedd00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedea060640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fede985f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedea060640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44762/0 (socket says 172.21.15.64:44762) 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedea060640 1 -- 172.21.15.64:0/2624231987 learned_addr learned my addr 172.21.15.64:0/2624231987 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fede985f640 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x561136352510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fede985f640 1 -- 172.21.15.64:0/2624231987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 msgr2=0x561136352510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fede985f640 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x561136352510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fede985f640 1 -- 172.21.15.64:0/2624231987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 msgr2=0x56113635cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fede985f640 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fede985f640 1 -- 172.21.15.64:0/2624231987 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561136029b90 con 0x56113635d310 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedea060640 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7feddb7fe640 1 -- 172.21.15.64:0/2351199358 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7feddc0f05c0 con 0x7fede4108030 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.210+0000 7fedea861640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fedd00a2cf0 0x7fedd00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.211+0000 7fede985f640 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635f700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009730 tx=0x7fedd40096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1f65cf702cd0eba server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.211+0000 7fedd8dea640 1 -- 172.21.15.64:0/2624231987 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd400fce0 con 0x56113635d310 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.211+0000 7fedea861640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fedd00a2cf0 0x7fedd00a51b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feddc0edb20 tx=0x7feddc005c30 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.211+0000 7fedd8dea640 1 -- 172.21.15.64:0/2624231987 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd400fe80 con 0x56113635d310 2024-08-19T01:36:08.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.211+0000 7fedd8dea640 1 -- 172.21.15.64:0/2624231987 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4004a00 con 0x56113635d310 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 -- 172.21.15.64:0/2624231987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 msgr2=0x56113635f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635f700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009730 tx=0x7fedd40096c0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 -- 172.21.15.64:0/2624231987 shutdown_connections 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635f700 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 --2- 172.21.15.64:0/2624231987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x561136352510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 -- 172.21.15.64:0/2624231987 >> 172.21.15.64:0/2624231987 conn(0x56113634e6c0 msgr2=0x56113634eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 -- 172.21.15.64:0/2624231987 shutdown_connections 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 -- 172.21.15.64:0/2624231987 wait complete. 2024-08-19T01:36:08.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 -- start start 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.212+0000 7fedee6c2740 1 Processor -- start 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113635a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedee6c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56113635fee0 con 0x561136352a50 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedee6c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561136272990 con 0x561136352130 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedee6c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561136360060 con 0x56113635d310 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedee6c2740 10 client.0 _ll_register_callbacks cb 0x561135fc7090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedea060640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fede985f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fede985f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:51060/0 (socket says 172.21.15.64:51060) 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fede985f640 1 -- 172.21.15.64:0/3308062416 learned_addr learned my addr 172.21.15.64:0/3308062416 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fede985f640 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113635a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedea060640 1 -- 172.21.15.64:0/3308062416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 msgr2=0x56113635acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedea060640 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedea060640 1 -- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 msgr2=0x56113635a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedea060640 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113635a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.213+0000 7fedea060640 1 -- 172.21.15.64:0/3308062416 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611362716c0 con 0x561136352a50 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedea060640 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635a7b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fede00126b0 tx=0x7fede0012b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddebde81590e0a0f server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedcdffb640 1 -- 172.21.15.64:0/3308062416 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede0004020 con 0x561136352a50 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5611362c51a0 con 0x561136352a50 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedcdffb640 1 -- 172.21.15.64:0/3308062416 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fede00041c0 con 0x561136352a50 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedcdffb640 1 -- 172.21.15.64:0/3308062416 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fede0025050 con 0x561136352a50 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedee6c2740 10 client.15585 fetch_fsmap learned FSMap version 27 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5611363744d0 con 0x561136352a50 2024-08-19T01:36:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.214+0000 7fedcdffb640 1 -- 172.21.15.64:0/3308062416 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5611362c51a0 con 0x561136352a50 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedcdffb640 1 -- 172.21.15.64:0/3308062416 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fede00039f0 con 0x561136352a50 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedee6c2740 10 client.15585 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedee6c2740 10 client.15585 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedee6c2740 20 client.15585 populate_metadata read hostname 'smithi064' 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561136375a20 0x561136377e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedee6c2740 4 client.15585 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611362bf4f0 con 0x561136375a20 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fede985f640 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561136375a20 0x561136377e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fede985f640 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561136375a20 0x561136377e10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=444328c9994a335d server_cookie=5596f0f77af1ac6f in_seq=0 out_seq=0 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.215+0000 7fedcdffb640 10 client.15585 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedcdffb640 1 -- 172.21.15.64:0/3308062416 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5611362bf4f0 con 0x561136375a20 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedcdffb640 10 client.15585 handle_command_reply: tid=0 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 client.15585 shutdown 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 msgr2=0x56113635a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635a7b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fede00126b0 tx=0x7fede0012b70 comp rx=0 tx=0).stop 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 shutdown_connections 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113635acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113635a7b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561136375a20 0x561136377e10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3308062416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113635a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 >> 172.21.15.64:0/3308062416 conn(0x56113634e6c0 msgr2=0x5611363543f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedcdffb640 0 client.15585 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 shutdown_connections 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.216+0000 7fedee6c2740 1 -- 172.21.15.64:0/3308062416 wait complete. 2024-08-19T01:36:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.217+0000 7fedee6c2740 20 client.15585 trim_cache size 0 max 16384 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.234+0000 7fedee6c2740 1 Processor -- start 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.234+0000 7fedee6c2740 1 -- start start 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113634d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113637d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113637dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedee6c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fede41af420 con 0x561136352a50 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedee6c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fede41accf0 con 0x561136352130 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedee6c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fede41aab40 con 0x56113635d310 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedea060640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113637d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113634d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedea060640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113637d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44786/0 (socket says 172.21.15.64:44786) 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedea060640 1 -- 172.21.15.64:0/368829031 learned_addr learned my addr 172.21.15.64:0/368829031 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113637dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 -- 172.21.15.64:0/368829031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 msgr2=0x56113637dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113637dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 -- 172.21.15.64:0/368829031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 msgr2=0x56113637d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113637d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 -- 172.21.15.64:0/368829031 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611363742e0 con 0x561136352130 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedea060640 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113637d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fede985f640 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113634d5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fedd40057c0 tx=0x7fedd4012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b69a59d9c49b4f83 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.235+0000 7fedcffff640 1 -- 172.21.15.64:0/368829031 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4015460 con 0x561136352130 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedcffff640 1 -- 172.21.15.64:0/368829031 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd4015600 con 0x561136352130 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedcffff640 1 -- 172.21.15.64:0/368829031 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4023440 con 0x561136352130 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 -- 172.21.15.64:0/368829031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 msgr2=0x56113634d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113634d5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fedd40057c0 tx=0x7fedd4012040 comp rx=0 tx=0).stop 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 -- 172.21.15.64:0/368829031 shutdown_connections 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x56113637dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x56113637d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 --2- 172.21.15.64:0/368829031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x56113634d5e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 -- 172.21.15.64:0/368829031 >> 172.21.15.64:0/368829031 conn(0x56113634e6c0 msgr2=0x56113634ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 -- 172.21.15.64:0/368829031 shutdown_connections 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.236+0000 7fedee6c2740 1 -- 172.21.15.64:0/368829031 wait complete. 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 -- start start 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 Processor -- start 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x5611363572d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x561136357810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x561136357d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fede41ab1b0 con 0x561136352a50 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fede41ac620 con 0x561136352130 2024-08-19T01:36:08.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fede41ab880 con 0x56113635d310 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedea060640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x561136357810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedee6c2740 10 client.0 _ll_register_callbacks cb 0x7fede0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedea060640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x561136357810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44794/0 (socket says 172.21.15.64:44794) 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fedea060640 1 -- 172.21.15.64:0/3203999133 learned_addr learned my addr 172.21.15.64:0/3203999133 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fede985f640 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x5611363572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.237+0000 7fede985f640 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x561136357d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedea060640 1 -- 172.21.15.64:0/3203999133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 msgr2=0x561136357d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedea060640 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x561136357d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedea060640 1 -- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 msgr2=0x5611363572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedea060640 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x5611363572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedea060640 1 -- 172.21.15.64:0/3203999133 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611363df660 con 0x561136352a50 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedea060640 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x561136357810 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fede0000c00 tx=0x7fede0005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620187ed1d013e48 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:36:08.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedcdffb640 1 -- 172.21.15.64:0/3203999133 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede00041c0 con 0x561136352a50 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedcdffb640 1 -- 172.21.15.64:0/3203999133 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fede0018040 con 0x561136352a50 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd4015600 con 0x561136352a50 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedcdffb640 1 -- 172.21.15.64:0/3203999133 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fede0037050 con 0x561136352a50 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedee6c2740 10 client.15591 fetch_fsmap learned FSMap version 27 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.238+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5611363e0860 con 0x561136352a50 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedcdffb640 1 -- 172.21.15.64:0/3203999133 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd4015600 con 0x561136352a50 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedcdffb640 1 -- 172.21.15.64:0/3203999133 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fede0003a30 con 0x561136352a50 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedee6c2740 10 client.15591 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedee6c2740 10 client.15591 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedee6c2740 20 client.15591 populate_metadata read hostname 'smithi064' 2024-08-19T01:36:08.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5611363e10e0 0x5611363e3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:08.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedee6c2740 4 client.15591 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-08-19T01:36:08.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x56113623bec0 con 0x5611363e10e0 2024-08-19T01:36:08.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.239+0000 7fede985f640 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5611363e10e0 0x5611363e3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:08.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.240+0000 7fede985f640 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5611363e10e0 0x5611363e3500 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1c7e6ea8ef6791 server_cookie=49e06b9731eeb106 in_seq=0 out_seq=0 2024-08-19T01:36:08.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.240+0000 7fedcdffb640 10 client.15591 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.312+0000 7fedcdffb640 1 -- 172.21.15.64:0/3203999133 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56113623bec0 con 0x5611363e10e0 2024-08-19T01:36:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.312+0000 7fedcdffb640 10 client.15591 handle_command_reply: tid=0 2024-08-19T01:36:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.312+0000 7fedee6c2740 1 client.15591 shutdown 2024-08-19T01:36:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.312+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 msgr2=0x561136357810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.312+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x561136357810 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fede0000c00 tx=0x7fede0005650 comp rx=0 tx=0).stop 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 shutdown_connections 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56113635d310 0x561136357d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561136352a50 0x561136357810 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5611363e10e0 0x5611363e3500 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 --2- 172.21.15.64:0/3203999133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561136352130 0x5611363572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 >> 172.21.15.64:0/3203999133 conn(0x56113634e6c0 msgr2=0x56113634d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedcdffb640 0 client.15591 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 shutdown_connections 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 1 -- 172.21.15.64:0/3203999133 wait complete. 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedee6c2740 20 client.15591 trim_cache size 0 max 16384 2024-08-19T01:36:08.315 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 27, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 10, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "72d72d7d": { 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 10, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 150.0, 2024-08-19T01:36:08.316 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:08.317 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fedd00a2cf0 msgr2=0x7fedd00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fedd00a2cf0 0x7fedd00a51b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feddc0edb20 tx=0x7feddc005c30 comp rx=0 tx=0).stop 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 msgr2=0x7fede419cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede419cbf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7feddc009ca0 tx=0x7feddc02f350 comp rx=0 tx=0).stop 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 shutdown_connections 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fedd00a2cf0 0x7fedd00a51b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fede4108030 0x7fede419cbf0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede41034b0 0x7fede4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 --2- 172.21.15.64:0/2351199358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fede4102b00 0x7fede4198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.313+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 >> 172.21.15.64:0/2351199358 conn(0x7fede40fc500 msgr2=0x7fede41060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.314+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 shutdown_connections 2024-08-19T01:36:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:08.314+0000 7fedec2c4640 1 -- 172.21.15.64:0/2351199358 wait complete. 2024-08-19T01:36:08.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 13.86 MB/sec warmup 100 sec latency 69.169 ms 2024-08-19T01:36:09.142 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '72d72d7d', quiesce took 3.7 seconds. Will release after: 26.3 2024-08-19T01:36:09.378 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 13.72 MB/sec warmup 101 sec latency 1065.622 ms 2024-08-19T01:36:09.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:09 smithi123 ceph-mon[21829]: pgmap v90: 129 pgs: 129 active+clean; 131 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 265 op/s 2024-08-19T01:36:09.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:09 smithi161 ceph-mon[21757]: pgmap v90: 129 pgs: 129 active+clean; 131 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 265 op/s 2024-08-19T01:36:09.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:09 smithi064 ceph-mon[18515]: pgmap v90: 129 pgs: 129 active+clean; 131 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 265 op/s 2024-08-19T01:36:10.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 13.58 MB/sec warmup 102 sec latency 2065.729 ms 2024-08-19T01:36:11.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 13.45 MB/sec warmup 103 sec latency 3065.831 ms 2024-08-19T01:36:11.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:11 smithi123 ceph-mon[21829]: pgmap v91: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 362 op/s 2024-08-19T01:36:11.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:11 smithi161 ceph-mon[21757]: pgmap v91: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 362 op/s 2024-08-19T01:36:11.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:11 smithi064 ceph-mon[18515]: pgmap v91: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 362 op/s 2024-08-19T01:36:12.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 13.32 MB/sec warmup 104 sec latency 4065.938 ms 2024-08-19T01:36:13.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 13.20 MB/sec warmup 105 sec latency 5066.023 ms 2024-08-19T01:36:13.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:13 smithi161 ceph-mon[21757]: pgmap v92: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-08-19T01:36:13.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:13 smithi064 ceph-mon[18515]: pgmap v92: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-08-19T01:36:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:13 smithi123 ceph-mon[21829]: pgmap v92: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-08-19T01:36:14.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 13.07 MB/sec warmup 106 sec latency 6066.110 ms 2024-08-19T01:36:15.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.95 MB/sec warmup 107 sec latency 7066.170 ms 2024-08-19T01:36:15.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:15 smithi161 ceph-mon[21757]: pgmap v93: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-08-19T01:36:15.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:15 smithi064 ceph-mon[18515]: pgmap v93: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-08-19T01:36:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:15 smithi123 ceph-mon[21829]: pgmap v93: 129 pgs: 129 active+clean; 135 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 235 op/s 2024-08-19T01:36:16.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.83 MB/sec warmup 108 sec latency 8066.264 ms 2024-08-19T01:36:17.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.71 MB/sec warmup 109 sec latency 9066.372 ms 2024-08-19T01:36:17.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:17 smithi161 ceph-mon[21757]: pgmap v94: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-08-19T01:36:17.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:17 smithi064 ceph-mon[18515]: pgmap v94: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-08-19T01:36:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:17 smithi123 ceph-mon[21829]: pgmap v94: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-08-19T01:36:18.379 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.60 MB/sec warmup 110 sec latency 10066.487 ms 2024-08-19T01:36:19.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.48 MB/sec warmup 111 sec latency 11066.595 ms 2024-08-19T01:36:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:19 smithi161 ceph-mon[21757]: pgmap v95: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-08-19T01:36:19.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:19 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:36:19.967 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:19 smithi123 ceph-mon[21829]: pgmap v95: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-08-19T01:36:19.967 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:19 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:36:19.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:19 smithi064 ceph-mon[18515]: pgmap v95: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-08-19T01:36:19.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:19 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:36:20.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.37 MB/sec warmup 112 sec latency 12066.702 ms 2024-08-19T01:36:20.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:20 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:36:20.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:20 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:36:20.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:20 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:36:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:20 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:36:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:20 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:36:20.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:20 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:36:21.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:20 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:36:21.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:20 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:36:21.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:20 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:36:21.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.26 MB/sec warmup 113 sec latency 13066.756 ms 2024-08-19T01:36:21.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:21 smithi161 ceph-mon[21757]: pgmap v96: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-08-19T01:36:21.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:21 smithi064 ceph-mon[18515]: pgmap v96: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-08-19T01:36:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:21 smithi123 ceph-mon[21829]: pgmap v96: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-08-19T01:36:22.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.15 MB/sec warmup 114 sec latency 14066.882 ms 2024-08-19T01:36:23.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 12.05 MB/sec warmup 115 sec latency 15066.961 ms 2024-08-19T01:36:23.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:23 smithi161 ceph-mon[21757]: pgmap v97: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 KiB/s wr, 15 op/s 2024-08-19T01:36:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:23 smithi064 ceph-mon[18515]: pgmap v97: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 KiB/s wr, 15 op/s 2024-08-19T01:36:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:23 smithi123 ceph-mon[21829]: pgmap v97: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 KiB/s wr, 15 op/s 2024-08-19T01:36:24.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 11.94 MB/sec warmup 116 sec latency 16067.082 ms 2024-08-19T01:36:25.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 11.84 MB/sec warmup 117 sec latency 17067.161 ms 2024-08-19T01:36:25.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:25 smithi161 ceph-mon[21757]: pgmap v98: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 KiB/s wr, 15 op/s 2024-08-19T01:36:25.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:25 smithi064 ceph-mon[18515]: pgmap v98: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 KiB/s wr, 15 op/s 2024-08-19T01:36:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:25 smithi123 ceph-mon[21829]: pgmap v98: 129 pgs: 129 active+clean; 135 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 KiB/s wr, 15 op/s 2024-08-19T01:36:26.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 11.74 MB/sec warmup 118 sec latency 18067.274 ms 2024-08-19T01:36:27.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 11.64 MB/sec warmup 119 sec latency 19067.381 ms 2024-08-19T01:36:27.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:27 smithi161 ceph-mon[21757]: pgmap v99: 129 pgs: 129 active+clean; 134 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1021 KiB/s wr, 51 op/s 2024-08-19T01:36:27.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:27 smithi064 ceph-mon[18515]: pgmap v99: 129 pgs: 129 active+clean; 134 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1021 KiB/s wr, 51 op/s 2024-08-19T01:36:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:27 smithi123 ceph-mon[21829]: pgmap v99: 129 pgs: 129 active+clean; 134 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1021 KiB/s wr, 51 op/s 2024-08-19T01:36:29.380 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 1 sec latency 21067.536 ms 2024-08-19T01:36:29.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:29 smithi161 ceph-mon[21757]: pgmap v100: 129 pgs: 129 active+clean; 134 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1003 KiB/s wr, 43 op/s 2024-08-19T01:36:29.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:29 smithi064 ceph-mon[18515]: pgmap v100: 129 pgs: 129 active+clean; 134 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1003 KiB/s wr, 43 op/s 2024-08-19T01:36:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:29 smithi123 ceph-mon[21829]: pgmap v100: 129 pgs: 129 active+clean; 134 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1003 KiB/s wr, 43 op/s 2024-08-19T01:36:30.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 2 sec latency 22067.631 ms 2024-08-19T01:36:31.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 3 sec latency 23067.731 ms 2024-08-19T01:36:31.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:31 smithi161 ceph-mon[21757]: pgmap v101: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-08-19T01:36:31.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:31 smithi064 ceph-mon[18515]: pgmap v101: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-08-19T01:36:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:31 smithi123 ceph-mon[21829]: pgmap v101: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-08-19T01:36:32.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 4 sec latency 24067.834 ms 2024-08-19T01:36:33.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 5 sec latency 25067.933 ms 2024-08-19T01:36:33.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:33 smithi064 ceph-mon[18515]: pgmap v102: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.8 MiB/s wr, 72 op/s 2024-08-19T01:36:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:33 smithi123 ceph-mon[21829]: pgmap v102: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.8 MiB/s wr, 72 op/s 2024-08-19T01:36:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:33 smithi161 ceph-mon[21757]: pgmap v102: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.8 MiB/s wr, 72 op/s 2024-08-19T01:36:34.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 6 sec latency 26068.046 ms 2024-08-19T01:36:35.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 7 sec latency 27068.148 ms 2024-08-19T01:36:35.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:36:35.714 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:36:35.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:35 smithi064 ceph-mon[18515]: pgmap v103: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.8 MiB/s wr, 72 op/s 2024-08-19T01:36:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:35 smithi123 ceph-mon[21829]: pgmap v103: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.8 MiB/s wr, 72 op/s 2024-08-19T01:36:36.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 -- 172.21.15.64:0/2945327192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 msgr2=0x7fe7c4102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:36.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/2945327192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c4102e40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009ca0 tx=0x7fe7bc02a950 comp rx=0 tx=0).stop 2024-08-19T01:36:36.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 -- 172.21.15.64:0/2945327192 shutdown_connections 2024-08-19T01:36:36.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/2945327192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7c410be90 0x7fe7c410e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/2945327192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7c4103380 0x7fe7c410b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/2945327192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c4102e40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 -- 172.21.15.64:0/2945327192 >> 172.21.15.64:0/2945327192 conn(0x7fe7c40fc500 msgr2=0x7fe7c40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 -- 172.21.15.64:0/2945327192 shutdown_connections 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 -- 172.21.15.64:0/2945327192 wait complete. 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.168+0000 7fe7cadf8640 1 Processor -- start 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7cadf8640 1 -- start start 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7cadf8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7cadf8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7c4103380 0x7fe7c41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7cadf8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7c410be90 0x7fe7c419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7cadf8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c4111c90 con 0x7fe7c4102a60 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7cadf8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c4111b10 con 0x7fe7c410be90 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7cadf8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c4111e10 con 0x7fe7c4103380 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:36.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c41981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51210/0 (socket says 172.21.15.64:51210) 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 -- 172.21.15.64:0/1347354984 learned_addr learned my addr 172.21.15.64:0/1347354984 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c9395640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7c410be90 0x7fe7c419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c3fff640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7c4103380 0x7fe7c41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 -- 172.21.15.64:0/1347354984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7c4103380 msgr2=0x7fe7c41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7c4103380 0x7fe7c41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 -- 172.21.15.64:0/1347354984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7c410be90 msgr2=0x7fe7c419ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7c410be90 0x7fe7c419ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 -- 172.21.15.64:0/1347354984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c419d160 con 0x7fe7c4102a60 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c8b94640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c41981b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc005cc0 tx=0x7fe7bc02f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.169+0000 7fe7c1ffb640 1 -- 172.21.15.64:0/1347354984 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc034040 con 0x7fe7c4102a60 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.170+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c419d3f0 con 0x7fe7c4102a60 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.170+0000 7fe7c1ffb640 1 -- 172.21.15.64:0/1347354984 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7bc030920 con 0x7fe7c4102a60 2024-08-19T01:36:36.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.170+0000 7fe7c1ffb640 1 -- 172.21.15.64:0/1347354984 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc004830 con 0x7fe7c4102a60 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.170+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c41a4d30 con 0x7fe7c4102a60 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.171+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c406ada0 con 0x7fe7c4102a60 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.172+0000 7fe7c1ffb640 1 -- 172.21.15.64:0/1347354984 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe7bc037020 con 0x7fe7c4102a60 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.172+0000 7fe7c1ffb640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe7ac0a2cf0 0x7fe7ac0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.173+0000 7fe7c3fff640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe7ac0a2cf0 0x7fe7ac0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.173+0000 7fe7c1ffb640 1 -- 172.21.15.64:0/1347354984 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe7bc0f06d0 con 0x7fe7c4102a60 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.173+0000 7fe7c3fff640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe7ac0a2cf0 0x7fe7ac0a51b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80058c0 tx=0x7fe7b8005850 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.177+0000 7fe7c1ffb640 1 -- 172.21.15.64:0/1347354984 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fe7bc0bd050 con 0x7fe7c4102a60 2024-08-19T01:36:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:35 smithi161 ceph-mon[21757]: pgmap v103: 129 pgs: 129 active+clean; 132 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.8 MiB/s wr, 72 op/s 2024-08-19T01:36:36.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 8 sec latency 28068.184 ms 2024-08-19T01:36:36.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.388+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7c4106f30 con 0x7fe7c4102a60 2024-08-19T01:36:36.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.389+0000 7fe7c1ffb640 1 -- 172.21.15.64:0/1347354984 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11924 (secure 0 0 0) 0x7fe7bc0bc950 con 0x7fe7c4102a60 2024-08-19T01:36:36.390 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:36:36.391 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"btime":"2024-08-19T01:34:36:714408+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:34:36.547976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:36:36.391 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2024-08-19T01:36:36.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe7ac0a2cf0 msgr2=0x7fe7ac0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:36.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe7ac0a2cf0 0x7fe7ac0a51b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80058c0 tx=0x7fe7b8005850 comp rx=0 tx=0).stop 2024-08-19T01:36:36.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 msgr2=0x7fe7c41981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:36.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c41981b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc005cc0 tx=0x7fe7bc02f630 comp rx=0 tx=0).stop 2024-08-19T01:36:36.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 shutdown_connections 2024-08-19T01:36:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe7ac0a2cf0 0x7fe7ac0a51b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7c410be90 0x7fe7c419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7c4103380 0x7fe7c41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 --2- 172.21.15.64:0/1347354984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7c4102a60 0x7fe7c41981b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 >> 172.21.15.64:0/1347354984 conn(0x7fe7c40fc500 msgr2=0x7fe7c40fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 shutdown_connections 2024-08-19T01:36:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:36.392+0000 7fe7cadf8640 1 -- 172.21.15.64:0/1347354984 wait complete. 2024-08-19T01:36:36.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:36 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1347354984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:37.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:36 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1347354984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:37.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:36 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1347354984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:37.205 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 72d72d7d --release --await' 2024-08-19T01:36:37.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 72d72d7d --release --await 2024-08-19T01:36:37.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 254229 0.00 MB/sec execute 9 sec latency 29068.221 ms 2024-08-19T01:36:37.465 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:36:37.883 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:37 smithi064 ceph-mon[18515]: pgmap v104: 129 pgs: 129 active+clean; 132 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-08-19T01:36:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 -- 172.21.15.64:0/4063909015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1078a0 msgr2=0x7fad1c107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 --2- 172.21.15.64:0/4063909015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1078a0 0x7fad1c107c80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fad18009930 tx=0x7fad1802a420 comp rx=0 tx=0).stop 2024-08-19T01:36:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 -- 172.21.15.64:0/4063909015 shutdown_connections 2024-08-19T01:36:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 --2- 172.21.15.64:0/4063909015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad1c102240 0x7fad1c10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 --2- 172.21.15.64:0/4063909015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad1c1018a0 0x7fad1c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 --2- 172.21.15.64:0/4063909015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1078a0 0x7fad1c107c80 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 -- 172.21.15.64:0/4063909015 >> 172.21.15.64:0/4063909015 conn(0x7fad1c0fd5e0 msgr2=0x7fad1c0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 -- 172.21.15.64:0/4063909015 shutdown_connections 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.969+0000 7fad234ca640 1 -- 172.21.15.64:0/4063909015 wait complete. 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.970+0000 7fad234ca640 1 Processor -- start 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.970+0000 7fad234ca640 1 -- start start 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.970+0000 7fad234ca640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1018a0 0x7fad1c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad234ca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad1c102240 0x7fad1c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad234ca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad1c1078a0 0x7fad1c19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad234ca640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c111ca0 con 0x7fad1c1018a0 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad234ca640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c111b20 con 0x7fad1c102240 2024-08-19T01:36:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad234ca640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c111e20 con 0x7fad1c1078a0 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1018a0 0x7fad1c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1018a0 0x7fad1c1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51238/0 (socket says 172.21.15.64:51238) 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 -- 172.21.15.64:0/1055883285 learned_addr learned my addr 172.21.15.64:0/1055883285 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21a67640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad1c1078a0 0x7fad1c19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad20a65640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad1c102240 0x7fad1c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 -- 172.21.15.64:0/1055883285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad1c1078a0 msgr2=0x7fad1c19cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad1c1078a0 0x7fad1c19cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 -- 172.21.15.64:0/1055883285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad1c102240 msgr2=0x7fad1c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad1c102240 0x7fad1c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 -- 172.21.15.64:0/1055883285 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c19d190 con 0x7fad1c1018a0 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad21266640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1018a0 0x7fad1c1981e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fad18009230 tx=0x7fad18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad127fc640 1 -- 172.21.15.64:0/1055883285 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1803f030 con 0x7fad1c1018a0 2024-08-19T01:36:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad127fc640 1 -- 172.21.15.64:0/1055883285 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad18003ad0 con 0x7fad1c1018a0 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1c19d420 con 0x7fad1c1018a0 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad127fc640 1 -- 172.21.15.64:0/1055883285 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18035070 con 0x7fad1c1018a0 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.971+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1c1a4d60 con 0x7fad1c1018a0 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.974+0000 7fad127fc640 1 -- 172.21.15.64:0/1055883285 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fad18042030 con 0x7fad1c1018a0 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.974+0000 7fad258c8740 1 Processor -- start 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad258c8740 1 -- start start 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad258c8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad258c8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9ad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad258c8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad258c8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559c6d908c60 con 0x559c6d9a32b0 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad258c8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559c6d8a0b90 con 0x559c6d9a2990 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad258c8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c6d904c20 con 0x559c6d9adb70 2024-08-19T01:36:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9ad630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51250/0 (socket says 172.21.15.64:51250) 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 -- 172.21.15.64:0/3545550876 learned_addr learned my addr 172.21.15.64:0/3545550876 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad20a65640 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21a67640 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 -- 172.21.15.64:0/3545550876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 msgr2=0x559c6d9aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 -- 172.21.15.64:0/3545550876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 msgr2=0x559c6d9a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad127fc640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fad080a2d70 0x7fad080a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.975+0000 7fad21266640 1 -- 172.21.15.64:0/3545550876 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c6d618580 con 0x559c6d9a32b0 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.976+0000 7fad21a67640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fad080a2d70 0x7fad080a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.976+0000 7fad21266640 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9ad630 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fad180ecef0 tx=0x7fad180f3eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2220633c705df7a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.976+0000 7fad127fc640 1 -- 172.21.15.64:0/1055883285 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fad180efbe0 con 0x7fad1c1018a0 2024-08-19T01:36:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.976+0000 7fad077fe640 1 -- 172.21.15.64:0/3545550876 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18049020 con 0x559c6d9a32b0 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.976+0000 7fad077fe640 1 -- 172.21.15.64:0/3545550876 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad180bbe00 con 0x559c6d9a32b0 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.976+0000 7fad077fe640 1 -- 172.21.15.64:0/3545550876 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad180bb940 con 0x559c6d9a32b0 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 -- 172.21.15.64:0/3545550876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 msgr2=0x559c6d9ad630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9ad630 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fad180ecef0 tx=0x7fad180f3eb0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad21a67640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fad080a2d70 0x7fad080a5230 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fad14005fd0 tx=0x7fad14009600 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 -- 172.21.15.64:0/3545550876 shutdown_connections 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9ad630 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 --2- 172.21.15.64:0/3545550876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 -- 172.21.15.64:0/3545550876 >> 172.21.15.64:0/3545550876 conn(0x559c6d99ef20 msgr2=0x559c6d99f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 -- 172.21.15.64:0/3545550876 shutdown_connections 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 -- 172.21.15.64:0/3545550876 wait complete. 2024-08-19T01:36:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 -- start start 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.977+0000 7fad258c8740 1 Processor -- start 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a32b0 0x559c6d9aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9adb70 0x559c6d9ab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559c6d9b0500 con 0x559c6d9adb70 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559c6d8a67c0 con 0x559c6d9a32b0 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c6d9b0680 con 0x559c6d9a2990 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 10 client.0 _ll_register_callbacks cb 0x559c6d8aff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9adb70 0x559c6d9ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9adb70 0x559c6d9ab460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51258/0 (socket says 172.21.15.64:51258) 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 -- 172.21.15.64:0/3189507931 learned_addr learned my addr 172.21.15.64:0/3189507931 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21266640 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad20a65640 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a32b0 0x559c6d9aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 -- 172.21.15.64:0/3189507931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 msgr2=0x559c6d9aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9aa9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 -- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a32b0 msgr2=0x559c6d9aaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a32b0 0x559c6d9aaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 -- 172.21.15.64:0/3189507931 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c6d6e2960 con 0x559c6d9adb70 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad21a67640 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9adb70 0x559c6d9ab460 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fad14011d70 tx=0x7fad14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327aad405e1ecd39 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad04ff9640 1 -- 172.21.15.64:0/3189507931 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14004d20 con 0x559c6d9adb70 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad04ff9640 1 -- 172.21.15.64:0/3189507931 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fad14005020 con 0x559c6d9adb70 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.978+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad180bbe00 con 0x559c6d9adb70 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad04ff9640 1 -- 172.21.15.64:0/3189507931 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fad14011e60 con 0x559c6d9adb70 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 10 client.15609 fetch_fsmap learned FSMap version 27 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c6d9bc490 con 0x559c6d9adb70 2024-08-19T01:36:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad04ff9640 1 -- 172.21.15.64:0/3189507931 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad180bbe00 con 0x559c6d9adb70 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad04ff9640 1 -- 172.21.15.64:0/3189507931 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fad1400f040 con 0x559c6d9adb70 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 10 client.15609 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 10 client.15609 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 20 client.15609 populate_metadata read hostname 'smithi064' 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6d9bda40 0x559c6d9bfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 4 client.15609 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c6d9161d0 con 0x559c6d9bda40 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.979+0000 7fad21266640 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6d9bda40 0x559c6d9bfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.980+0000 7fad04ff9640 10 client.15609 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.980+0000 7fad21266640 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6d9bda40 0x559c6d9bfe80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e07aac3ca99c7762 server_cookie=e9b753f95efab47e in_seq=0 out_seq=0 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.980+0000 7fad04ff9640 1 -- 172.21.15.64:0/3189507931 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559c6d9161d0 con 0x559c6d9bda40 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.980+0000 7fad04ff9640 10 client.15609 handle_command_reply: tid=0 2024-08-19T01:36:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 client.15609 shutdown 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9adb70 msgr2=0x559c6d9ab460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9adb70 0x559c6d9ab460 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fad14011d70 tx=0x7fad14004a40 comp rx=0 tx=0).stop 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 shutdown_connections 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9adb70 0x559c6d9ab460 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6d9bda40 0x559c6d9bfe80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a32b0 0x559c6d9aaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 --2- 172.21.15.64:0/3189507931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 >> 172.21.15.64:0/3189507931 conn(0x559c6d99ef20 msgr2=0x559c6d9a4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad04ff9640 0 client.15609 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 shutdown_connections 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 1 -- 172.21.15.64:0/3189507931 wait complete. 2024-08-19T01:36:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.981+0000 7fad258c8740 20 client.15609 trim_cache size 0 max 16384 2024-08-19T01:36:38.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 Processor -- start 2024-08-19T01:36:38.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 -- start start 2024-08-19T01:36:38.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:38.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:38.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9adb70 0x559c6d9c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c1b1710 con 0x559c6d9a32b0 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c1aefe0 con 0x559c6d9adb70 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.998+0000 7fad258c8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c1ace30 con 0x559c6d9a2990 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9c5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51268/0 (socket says 172.21.15.64:51268) 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 -- 172.21.15.64:0/3393137644 learned_addr learned my addr 172.21.15.64:0/3393137644 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad21a67640 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9adb70 0x559c6d9c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad21266640 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 -- 172.21.15.64:0/3393137644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 msgr2=0x559c6d9a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 -- 172.21.15.64:0/3393137644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9adb70 msgr2=0x559c6d9c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9adb70 0x559c6d9c5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 -- 172.21.15.64:0/3393137644 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c6d99ed50 con 0x559c6d9a32b0 2024-08-19T01:36:38.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad21a67640 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9adb70 0x559c6d9c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:36:38.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad20a65640 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9c5790 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fad0c005d00 tx=0x7fad0c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d9a58abca37563 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:38.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad06ffd640 1 -- 172.21.15.64:0/3393137644 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c004020 con 0x559c6d9a32b0 2024-08-19T01:36:38.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad06ffd640 1 -- 172.21.15.64:0/3393137644 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad0c005500 con 0x559c6d9a32b0 2024-08-19T01:36:38.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:37.999+0000 7fad06ffd640 1 -- 172.21.15.64:0/3393137644 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c004d10 con 0x559c6d9a32b0 2024-08-19T01:36:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 -- 172.21.15.64:0/3393137644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 msgr2=0x559c6d9c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9c5790 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fad0c005d00 tx=0x7fad0c011810 comp rx=0 tx=0).stop 2024-08-19T01:36:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 -- 172.21.15.64:0/3393137644 shutdown_connections 2024-08-19T01:36:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9adb70 0x559c6d9c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9c5790 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 --2- 172.21.15.64:0/3393137644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9a2990 0x559c6d9a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 -- 172.21.15.64:0/3393137644 >> 172.21.15.64:0/3393137644 conn(0x559c6d99ef20 msgr2=0x559c6d9aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 -- 172.21.15.64:0/3393137644 shutdown_connections 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 -- 172.21.15.64:0/3393137644 wait complete. 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.000+0000 7fad258c8740 1 -- start start 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 1 Processor -- start 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9b1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c1ad4a0 con 0x559c6d9a32b0 2024-08-19T01:36:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c1ae910 con 0x559c6d9a2990 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c1adb70 con 0x559c6d9adb70 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad258c8740 10 client.0 _ll_register_callbacks cb 0x7fad140119f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51278/0 (socket says 172.21.15.64:51278) 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 -- 172.21.15.64:0/3654745249 learned_addr learned my addr 172.21.15.64:0/3654745249 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad21266640 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad21a67640 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 -- 172.21.15.64:0/3654745249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 msgr2=0x559c6d9b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 -- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 msgr2=0x559c6d9b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.001+0000 7fad20a65640 1 -- 172.21.15.64:0/3654745249 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c6da27960 con 0x559c6d9a32b0 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad20a65640 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9b1b70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fad0c006010 tx=0x7fad0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4245453e9cb162 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:36:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad04ff9640 1 -- 172.21.15.64:0/3654745249 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c02b020 con 0x559c6d9a32b0 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad0c005500 con 0x559c6d9a32b0 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad04ff9640 1 -- 172.21.15.64:0/3654745249 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fad0c0052b0 con 0x559c6d9a32b0 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad04ff9640 1 -- 172.21.15.64:0/3654745249 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fad0c036050 con 0x559c6d9a32b0 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 10 client.15615 fetch_fsmap learned FSMap version 27 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c6da28c40 con 0x559c6d9a32b0 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad04ff9640 1 -- 172.21.15.64:0/3654745249 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad0c005500 con 0x559c6d9a32b0 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad04ff9640 1 -- 172.21.15.64:0/3654745249 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fad0c0324b0 con 0x559c6d9a32b0 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 10 client.15615 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 10 client.15615 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 20 client.15615 populate_metadata read hostname 'smithi064' 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6da29390 0x559c6da2b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 4 client.15615 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "72d72d7d", "release": true, "await": true}] 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.002+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "72d72d7d", "release": true, "await": true}) -- 0x559c6d99a1d0 con 0x559c6da29390 2024-08-19T01:36:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.003+0000 7fad21266640 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6da29390 0x559c6da2b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:38.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.003+0000 7fad21266640 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6da29390 0x559c6da2b7b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c939a2a828e85b75 server_cookie=64b765d8f6232b48 in_seq=0 out_seq=0 2024-08-19T01:36:38.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.003+0000 7fad04ff9640 10 client.15615 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:38.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad04ff9640 1 -- 172.21.15.64:0/3654745249 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559c6d99a1d0 con 0x559c6da29390 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad04ff9640 10 client.15615 handle_command_reply: tid=0 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 client.15615 shutdown 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 msgr2=0x559c6d9b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9b1b70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fad0c006010 tx=0x7fad0c0046c0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 shutdown_connections 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c6d9adb70 0x559c6d9b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559c6d9a32b0 0x559c6d9b1b70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559c6da29390 0x559c6da2b7b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 --2- 172.21.15.64:0/3654745249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c6d9a2990 0x559c6d9b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 >> 172.21.15.64:0/3654745249 conn(0x559c6d99ef20 msgr2=0x559c6d9aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad04ff9640 0 client.15615 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 shutdown_connections 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 1 -- 172.21.15.64:0/3654745249 wait complete. 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.011+0000 7fad258c8740 20 client.15615 trim_cache size 0 max 16384 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 27, 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 12, 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "72d72d7d": { 2024-08-19T01:36:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 12, 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 150.0, 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:36:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:38.017 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:36:38.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fad080a2d70 msgr2=0x7fad080a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fad080a2d70 0x7fad080a5230 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fad14005fd0 tx=0x7fad14009600 comp rx=0 tx=0).stop 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1018a0 msgr2=0x7fad1c1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1018a0 0x7fad1c1981e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fad18009230 tx=0x7fad18004270 comp rx=0 tx=0).stop 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 shutdown_connections 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fad080a2d70 0x7fad080a5230 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad1c1078a0 0x7fad1c19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad1c102240 0x7fad1c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 --2- 172.21.15.64:0/1055883285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad1c1018a0 0x7fad1c1981e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 >> 172.21.15.64:0/1055883285 conn(0x7fad1c0fd5e0 msgr2=0x7fad1c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 shutdown_connections 2024-08-19T01:36:38.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:38.012+0000 7fad234ca640 1 -- 172.21.15.64:0/1055883285 wait complete. 2024-08-19T01:36:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:37 smithi123 ceph-mon[21829]: pgmap v104: 129 pgs: 129 active+clean; 132 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-08-19T01:36:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:37 smithi161 ceph-mon[21757]: pgmap v104: 129 pgs: 129 active+clean; 132 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-08-19T01:36:38.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 255085 0.64 MB/sec execute 10 sec latency 29694.731 ms 2024-08-19T01:36:38.900 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '72d72d7d', total seconds elapsed: 33.4 2024-08-19T01:36:38.900 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-08-19T01:36:38.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:36:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:36:39.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 257030 1.58 MB/sec execute 11 sec latency 15.928 ms 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 -- 172.21.15.64:0/2139029257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4980102c20 msgr2=0x7f498010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 --2- 172.21.15.64:0/2139029257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4980102c20 0x7f498010d770 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4978009ca0 tx=0x7f497802a950 comp rx=0 tx=0).stop 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 -- 172.21.15.64:0/2139029257 shutdown_connections 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 --2- 172.21.15.64:0/2139029257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4980102c20 0x7f498010d770 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 --2- 172.21.15.64:0/2139029257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49801022a0 0x7f49801026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 --2- 172.21.15.64:0/2139029257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980100b50 0x7f4980100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 -- 172.21.15.64:0/2139029257 >> 172.21.15.64:0/2139029257 conn(0x7f49800fc500 msgr2=0x7f49800fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 -- 172.21.15.64:0/2139029257 shutdown_connections 2024-08-19T01:36:39.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.678+0000 7f498840c640 1 -- 172.21.15.64:0/2139029257 wait complete. 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 Processor -- start 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 -- start start 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980100b50 0x7f49801981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49801022a0 0x7f49801986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980102c20 0x7f498019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4980111dc0 con 0x7f49801022a0 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4980111c40 con 0x7f4980102c20 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f498840c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4980111f40 con 0x7f4980100b50 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f49859a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49801022a0 0x7f49801986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f49861a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980100b50 0x7f49801981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f49869a9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980102c20 0x7f498019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f49859a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49801022a0 0x7f49801986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41050/0 (socket says 172.21.15.64:41050) 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f49859a7640 1 -- 172.21.15.64:0/829907095 learned_addr learned my addr 172.21.15.64:0/829907095 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:39.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.679+0000 7f49861a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980100b50 0x7f49801981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:42648/0 (socket says 172.21.15.64:42648) 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49869a9640 1 -- 172.21.15.64:0/829907095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980100b50 msgr2=0x7f49801981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49869a9640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980100b50 0x7f49801981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49869a9640 1 -- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49801022a0 msgr2=0x7f49801986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49869a9640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49801022a0 0x7f49801986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49869a9640 1 -- 172.21.15.64:0/829907095 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498019d150 con 0x7f4980102c20 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49861a8640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980100b50 0x7f49801981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49859a7640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49801022a0 0x7f49801986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49869a9640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980102c20 0x7f498019ca70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4978005cc0 tx=0x7f497802f3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49737fe640 1 -- 172.21.15.64:0/829907095 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978034040 con 0x7f4980102c20 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498019d3e0 con 0x7f4980102c20 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49737fe640 1 -- 172.21.15.64:0/829907095 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4978030920 con 0x7f4980102c20 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f49737fe640 1 -- 172.21.15.64:0/829907095 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49780045b0 con 0x7f4980102c20 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.680+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49801a4d20 con 0x7f4980102c20 2024-08-19T01:36:39.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.681+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4980100f30 con 0x7f4980102c20 2024-08-19T01:36:39.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.682+0000 7f49737fe640 1 -- 172.21.15.64:0/829907095 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4978037020 con 0x7f4980102c20 2024-08-19T01:36:39.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.683+0000 7f49737fe640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f496c0a2cf0 0x7f496c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:39.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.683+0000 7f49861a8640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f496c0a2cf0 0x7f496c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:39.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.683+0000 7f49737fe640 1 -- 172.21.15.64:0/829907095 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f49780f0480 con 0x7f4980102c20 2024-08-19T01:36:39.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.683+0000 7f49861a8640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f496c0a2cf0 0x7f496c0a51b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f497c009fc0 tx=0x7f497c005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:39.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.686+0000 7f49737fe640 1 -- 172.21.15.64:0/829907095 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f49780bcda0 con 0x7f4980102c20 2024-08-19T01:36:39.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.911+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49801a4ef0 con 0x7f4980102c20 2024-08-19T01:36:39.914 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:39 smithi064 ceph-mon[18515]: pgmap v105: 129 pgs: 129 active+clean; 132 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.0 MiB/s wr, 74 op/s 2024-08-19T01:36:39.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.911+0000 7f49737fe640 1 -- 172.21.15.64:0/829907095 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11924 (secure 0 0 0) 0x7f49780098b0 con 0x7f4980102c20 2024-08-19T01:36:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:36:39.914 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"btime":"2024-08-19T01:34:36:714408+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:34:36.547976+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:36:39.915 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f496c0a2cf0 msgr2=0x7f496c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f496c0a2cf0 0x7f496c0a51b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f497c009fc0 tx=0x7f497c005f90 comp rx=0 tx=0).stop 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980102c20 msgr2=0x7f498019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980102c20 0x7f498019ca70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4978005cc0 tx=0x7f497802f3a0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 shutdown_connections 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f496c0a2cf0 0x7f496c0a51b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980102c20 0x7f498019ca70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49801022a0 0x7f49801986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 --2- 172.21.15.64:0/829907095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980100b50 0x7f49801981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:39.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 >> 172.21.15.64:0/829907095 conn(0x7f49800fc500 msgr2=0x7f4980106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:39.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 shutdown_connections 2024-08-19T01:36:39.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:39.914+0000 7f498840c640 1 -- 172.21.15.64:0/829907095 wait complete. 2024-08-19T01:36:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:39 smithi123 ceph-mon[21829]: pgmap v105: 129 pgs: 129 active+clean; 132 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.0 MiB/s wr, 74 op/s 2024-08-19T01:36:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:39 smithi161 ceph-mon[21757]: pgmap v105: 129 pgs: 129 active+clean; 132 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.0 MiB/s wr, 74 op/s 2024-08-19T01:36:40.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 259782 2.65 MB/sec execute 12 sec latency 37.824 ms 2024-08-19T01:36:40.721 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-08-19T01:36:40.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-08-19T01:36:40.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:40 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/829907095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:40.990 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:36:41.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:40 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/829907095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:41.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:40 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/829907095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:36:41.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 262992 3.88 MB/sec execute 13 sec latency 12.801 ms 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 -- 172.21.15.64:0/897597178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41067b0 msgr2=0x7f0ed4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/897597178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41067b0 0x7f0ed4106b90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009930 tx=0x7f0ec402a420 comp rx=0 tx=0).stop 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 -- 172.21.15.64:0/897597178 shutdown_connections 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/897597178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ed4101150 0x7f0ed410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/897597178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ed41007b0 0x7f0ed4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/897597178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41067b0 0x7f0ed4106b90 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 -- 172.21.15.64:0/897597178 >> 172.21.15.64:0/897597178 conn(0x7f0ed40fc4d0 msgr2=0x7f0ed40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 -- 172.21.15.64:0/897597178 shutdown_connections 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.470+0000 7f0ed8cab640 1 -- 172.21.15.64:0/897597178 wait complete. 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 Processor -- start 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 -- start start 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41007b0 0x7f0ed40fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ed4101150 0x7f0ed40fcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ed41067b0 0x7f0ed40fd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4110c00 con 0x7f0ed41007b0 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4110a80 con 0x7f0ed4101150 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed8cab640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4110d80 con 0x7f0ed41067b0 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41007b0 0x7f0ed40fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41007b0 0x7f0ed40fcaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41066/0 (socket says 172.21.15.64:41066) 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed259c640 1 -- 172.21.15.64:0/704351814 learned_addr learned my addr 172.21.15.64:0/704351814 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ed41067b0 0x7f0ed40fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.471+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ed4101150 0x7f0ed40fcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed259c640 1 -- 172.21.15.64:0/704351814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ed41067b0 msgr2=0x7f0ed40fd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed259c640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ed41067b0 0x7f0ed40fd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed259c640 1 -- 172.21.15.64:0/704351814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ed4101150 msgr2=0x7f0ed40fcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed259c640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ed4101150 0x7f0ed40fcfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed259c640 1 -- 172.21.15.64:0/704351814 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed41a6640 con 0x7f0ed41007b0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ed41067b0 0x7f0ed40fd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed259c640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41007b0 0x7f0ed40fcaa0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0ec402a940 tx=0x7f0ec4003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ec37fe640 1 -- 172.21.15.64:0/704351814 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec403f070 con 0x7f0ed41007b0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ec37fe640 1 -- 172.21.15.64:0/704351814 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec4031c90 con 0x7f0ed41007b0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed41a68d0 con 0x7f0ed41007b0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ec37fe640 1 -- 172.21.15.64:0/704351814 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4035030 con 0x7f0ed41007b0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.472+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed41a6e00 con 0x7f0ed41007b0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0ec37fe640 1 -- 172.21.15.64:0/704351814 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0ec4042070 con 0x7f0ed41007b0 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 Processor -- start 2024-08-19T01:36:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 -- start start 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360130 0x55d92c360510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360a50 0x55d92c36add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c36d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d92c2c5a70 con 0x55d92c360a50 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55d92c25d780 con 0x55d92c36b310 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d92c2c1a30 con 0x55d92c360130 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.475+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360a50 0x55d92c36add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360130 0x55d92c360510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360a50 0x55d92c36add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41080/0 (socket says 172.21.15.64:41080) 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 -- 172.21.15.64:0/2313355656 learned_addr learned my addr 172.21.15.64:0/2313355656 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c36d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ec37fe640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0ebc0a2d70 0x7f0ebc0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 -- 172.21.15.64:0/2313355656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360130 msgr2=0x55d92c360510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360130 0x55d92c360510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 -- 172.21.15.64:0/2313355656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 msgr2=0x55d92c36d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c36d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 -- 172.21.15.64:0/2313355656 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d92c037b90 con 0x55d92c360a50 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360130 0x55d92c360510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c36d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0ebc0a2d70 0x7f0ebc0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ec37fe640 1 -- 172.21.15.64:0/704351814 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0ec40efe30 con 0x7f0ed41007b0 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed259c640 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360a50 0x55d92c36add0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40ed140 tx=0x7f0ec4086bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5b60f201099e7b server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ec0ff9640 1 -- 172.21.15.64:0/2313355656 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec40bb900 con 0x55d92c360a50 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0ebc0a2d70 0x7f0ebc0a5230 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc005c40 tx=0x7f0ecc004a40 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ec0ff9640 1 -- 172.21.15.64:0/2313355656 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec4031e40 con 0x55d92c360a50 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.476+0000 7f0ec0ff9640 1 -- 172.21.15.64:0/2313355656 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec408e210 con 0x55d92c360a50 2024-08-19T01:36:41.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2313355656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360a50 msgr2=0x55d92c36add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360a50 0x55d92c36add0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40ed140 tx=0x7f0ec4086bd0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2313355656 shutdown_connections 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c36d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360a50 0x55d92c36add0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2313355656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360130 0x55d92c360510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2313355656 >> 172.21.15.64:0/2313355656 conn(0x55d92c35c6c0 msgr2=0x55d92c35caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2313355656 shutdown_connections 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.477+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2313355656 wait complete. 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 -- start start 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 Processor -- start 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c368110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360a50 0x55d92c368650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c368b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d92c36dca0 con 0x55d92c360130 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55d92c280990 con 0x55d92c36b310 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d92c36de20 con 0x55d92c360a50 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0edb0a9740 10 client.0 _ll_register_callbacks cb 0x55d92bfd5090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c368110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c368110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41084/0 (socket says 172.21.15.64:41084) 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0ed259c640 1 -- 172.21.15.64:0/627863687 learned_addr learned my addr 172.21.15.64:0/627863687 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360a50 0x55d92c368650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c368b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0ed259c640 1 -- 172.21.15.64:0/627863687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360a50 msgr2=0x55d92c368650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.478+0000 7f0ed259c640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360a50 0x55d92c368650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0ed259c640 1 -- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 msgr2=0x55d92c368b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0ed259c640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c368b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0ed259c640 1 -- 172.21.15.64:0/627863687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d92c27f6c0 con 0x55d92c360130 2024-08-19T01:36:41.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360a50 0x55d92c368650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0ed259c640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c368110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40bbf30 tx=0x7f0ec4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9127f7dd6d4cf2 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0eb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/627863687 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4049070 con 0x55d92c360130 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/627863687 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0ec40ba2a0 con 0x55d92c360130 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d92c2d31a0 con 0x55d92c360130 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/627863687 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ec408ee70 con 0x55d92c360130 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0edb0a9740 10 client.15633 fetch_fsmap learned FSMap version 27 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d92c36dca0 con 0x55d92c360130 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.479+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/627863687 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d92c2d31a0 con 0x55d92c360130 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/627863687 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0ec408ace0 con 0x55d92c360130 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0edb0a9740 10 client.15633 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0edb0a9740 10 client.15633 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0edb0a9740 20 client.15633 populate_metadata read hostname 'smithi064' 2024-08-19T01:36:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c37ab70 0x55d92c37cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0edb0a9740 4 client.15633 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d92c2cd4f0 con 0x55d92c37ab70 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c37ab70 0x55d92c37cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c37ab70 0x55d92c37cff0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2b94edd9cef7cc1 server_cookie=3e763a908e28f82 in_seq=0 out_seq=0 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.480+0000 7f0eb9ffb640 10 client.15633 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/627863687 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d92c2cd4f0 con 0x55d92c37ab70 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0eb9ffb640 10 client.15633 handle_command_reply: tid=0 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 client.15633 shutdown 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 msgr2=0x55d92c368110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c368110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40bbf30 tx=0x7f0ec4009260 comp rx=0 tx=0).stop 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 shutdown_connections 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c37ab70 0x55d92c37cff0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c36b310 0x55d92c368b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c360a50 0x55d92c368650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0eb9ffb640 0 client.15633 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/627863687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c368110 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 >> 172.21.15.64:0/627863687 conn(0x55d92c35c6c0 msgr2=0x55d92c3623f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 shutdown_connections 2024-08-19T01:36:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.481+0000 7f0edb0a9740 1 -- 172.21.15.64:0/627863687 wait complete. 2024-08-19T01:36:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.482+0000 7f0edb0a9740 20 client.15633 trim_cache size 0 max 16384 2024-08-19T01:36:41.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.501+0000 7f0edb0a9740 1 Processor -- start 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.501+0000 7f0edb0a9740 1 -- start start 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c35b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3828c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c382e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41b38a0 con 0x55d92c360130 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41b1170 con 0x55d92c360a50 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41aefc0 con 0x55d92c36b310 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c35b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c35b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41096/0 (socket says 172.21.15.64:41096) 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 -- 172.21.15.64:0/3623507808 learned_addr learned my addr 172.21.15.64:0/3623507808 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3828c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c382e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 -- 172.21.15.64:0/3623507808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 msgr2=0x55d92c382e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c382e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 -- 172.21.15.64:0/3623507808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 msgr2=0x55d92c3828c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3828c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 -- 172.21.15.64:0/3623507808 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d92c35ac70 con 0x55d92c360130 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3828c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ed259c640 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c35b5e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40b8040 tx=0x7f0ec403b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f237fb3aa9d6e38 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ebbfff640 1 -- 172.21.15.64:0/3623507808 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4049020 con 0x55d92c360130 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ebbfff640 1 -- 172.21.15.64:0/3623507808 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec40bc6a0 con 0x55d92c360130 2024-08-19T01:36:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.502+0000 7f0ebbfff640 1 -- 172.21.15.64:0/3623507808 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec408f510 con 0x55d92c360130 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 -- 172.21.15.64:0/3623507808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 msgr2=0x55d92c35b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c35b5e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40b8040 tx=0x7f0ec403b720 comp rx=0 tx=0).stop 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 -- 172.21.15.64:0/3623507808 shutdown_connections 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c382e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3828c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/3623507808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c35b5e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 -- 172.21.15.64:0/3623507808 >> 172.21.15.64:0/3623507808 conn(0x55d92c35c6c0 msgr2=0x55d92c35aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 -- 172.21.15.64:0/3623507808 shutdown_connections 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.503+0000 7f0edb0a9740 1 -- 172.21.15.64:0/3623507808 wait complete. 2024-08-19T01:36:41.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 -- start start 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 Processor -- start 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c373260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3737a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c364b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41af630 con 0x55d92c360130 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41b0aa0 con 0x55d92c360a50 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41afd00 con 0x55d92c36b310 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c373260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c373260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41100/0 (socket says 172.21.15.64:41100) 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed259c640 1 -- 172.21.15.64:0/2542632196 learned_addr learned my addr 172.21.15.64:0/2542632196 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c364b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0edb0a9740 10 client.0 _ll_register_callbacks cb 0x7f0ecc009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3737a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed2d9d640 1 -- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 msgr2=0x55d92c3737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3737a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.504+0000 7f0ed2d9d640 1 -- 172.21.15.64:0/2542632196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 msgr2=0x55d92c373260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c373260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0ed2d9d640 1 -- 172.21.15.64:0/2542632196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d92c365380 con 0x55d92c36b310 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0ed259c640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c373260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0ed1d9b640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0ed2d9d640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c364b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009fd0 tx=0x7f0ecc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b679f2252650203 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0eb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/2542632196 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc02e070 con 0x55d92c36b310 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/2542632196 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0ecc00e7c0 con 0x55d92c36b310 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ec40bc6a0 con 0x55d92c36b310 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/2542632196 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ecc03a050 con 0x55d92c36b310 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0edb0a9740 10 client.25099 fetch_fsmap learned FSMap version 27 2024-08-19T01:36:41.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d92c372c00 con 0x55d92c36b310 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.505+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/2542632196 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec40bc6a0 con 0x55d92c36b310 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/2542632196 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0ecc02ac00 con 0x55d92c36b310 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0edb0a9740 10 client.25099 fetch_fsmap finished waiting for FSMap version 27 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0edb0a9740 10 client.25099 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0edb0a9740 20 client.25099 populate_metadata read hostname 'smithi064' 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c3e5d20 0x55d92c3e8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0edb0a9740 4 client.25099 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55d92c249ec0 con 0x55d92c3e5d20 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0ed259c640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c3e5d20 0x55d92c3e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0eb9ffb640 10 client.25099 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:41.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.506+0000 7f0ed259c640 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c3e5d20 0x55d92c3e8140 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9959e60d54e88a8 server_cookie=17370181dbac4f0b in_seq=0 out_seq=0 2024-08-19T01:36:41.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.565+0000 7f0eb9ffb640 1 -- 172.21.15.64:0/2542632196 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d92c249ec0 con 0x55d92c3e5d20 2024-08-19T01:36:41.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.565+0000 7f0eb9ffb640 10 client.25099 handle_command_reply: tid=0 2024-08-19T01:36:41.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.565+0000 7f0edb0a9740 1 client.25099 shutdown 2024-08-19T01:36:41.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.565+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 msgr2=0x55d92c364b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c364b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009fd0 tx=0x7f0ecc004ab0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 shutdown_connections 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d92c36b310 0x55d92c364b50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d92c3e5d20 0x55d92c3e8140 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d92c360a50 0x55d92c3737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0eb9ffb640 0 client.25099 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 --2- 172.21.15.64:0/2542632196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92c360130 0x55d92c373260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 >> 172.21.15.64:0/2542632196 conn(0x55d92c35c6c0 msgr2=0x55d92c35b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 shutdown_connections 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 1 -- 172.21.15.64:0/2542632196 wait complete. 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0edb0a9740 20 client.25099 trim_cache size 0 max 16384 2024-08-19T01:36:41.569 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 27, 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 14, 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "4a41811a": { 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 14, 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:36:41.570 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 150.0, 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:36:41.571 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0ebc0a2d70 msgr2=0x7f0ebc0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0ebc0a2d70 0x7f0ebc0a5230 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc005c40 tx=0x7f0ecc004a40 comp rx=0 tx=0).stop 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41007b0 msgr2=0x7f0ed40fcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.566+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41007b0 0x7f0ed40fcaa0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0ec402a940 tx=0x7f0ec4003b20 comp rx=0 tx=0).stop 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 shutdown_connections 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0ebc0a2d70 0x7f0ebc0a5230 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ed41067b0 0x7f0ed40fd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ed4101150 0x7f0ed40fcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 --2- 172.21.15.64:0/704351814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ed41007b0 0x7f0ed40fcaa0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:36:41.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 >> 172.21.15.64:0/704351814 conn(0x7f0ed40fc4d0 msgr2=0x7f0ed406a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:36:41.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 shutdown_connections 2024-08-19T01:36:41.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:36:41.567+0000 7f0ed8cab640 1 -- 172.21.15.64:0/704351814 wait complete. 2024-08-19T01:36:41.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:41 smithi064 ceph-mon[18515]: pgmap v106: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 145 op/s 2024-08-19T01:36:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:41 smithi123 ceph-mon[21829]: pgmap v106: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 145 op/s 2024-08-19T01:36:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:41 smithi161 ceph-mon[21757]: pgmap v106: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 145 op/s 2024-08-19T01:36:42.381 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 3.62 MB/sec execute 14 sec latency 874.674 ms 2024-08-19T01:36:42.406 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '4a41811a', quiesce took 3.5 seconds. Will release after: 26.5 2024-08-19T01:36:43.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 3.38 MB/sec execute 15 sec latency 1874.780 ms 2024-08-19T01:36:43.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:43 smithi064 ceph-mon[18515]: pgmap v107: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-08-19T01:36:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:43 smithi123 ceph-mon[21829]: pgmap v107: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-08-19T01:36:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:43 smithi161 ceph-mon[21757]: pgmap v107: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-08-19T01:36:44.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 3.17 MB/sec execute 16 sec latency 2874.878 ms 2024-08-19T01:36:45.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.98 MB/sec execute 17 sec latency 3874.998 ms 2024-08-19T01:36:45.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:45 smithi064 ceph-mon[18515]: pgmap v108: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-08-19T01:36:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:45 smithi123 ceph-mon[21829]: pgmap v108: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-08-19T01:36:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:45 smithi161 ceph-mon[21757]: pgmap v108: 129 pgs: 129 active+clean; 133 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 109 op/s 2024-08-19T01:36:46.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.81 MB/sec execute 18 sec latency 4875.095 ms 2024-08-19T01:36:47.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.67 MB/sec execute 19 sec latency 5875.173 ms 2024-08-19T01:36:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:47 smithi064 ceph-mon[18515]: pgmap v109: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-08-19T01:36:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:47 smithi123 ceph-mon[21829]: pgmap v109: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-08-19T01:36:48.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:47 smithi161 ceph-mon[21757]: pgmap v109: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-08-19T01:36:48.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.53 MB/sec execute 20 sec latency 6875.277 ms 2024-08-19T01:36:49.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.41 MB/sec execute 21 sec latency 7875.377 ms 2024-08-19T01:36:49.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:49 smithi064 ceph-mon[18515]: pgmap v110: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 116 op/s 2024-08-19T01:36:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:49 smithi123 ceph-mon[21829]: pgmap v110: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 116 op/s 2024-08-19T01:36:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:49 smithi161 ceph-mon[21757]: pgmap v110: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 116 op/s 2024-08-19T01:36:50.382 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.30 MB/sec execute 22 sec latency 8875.483 ms 2024-08-19T01:36:51.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.20 MB/sec execute 23 sec latency 9875.584 ms 2024-08-19T01:36:51.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:51 smithi064 ceph-mon[18515]: pgmap v111: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-08-19T01:36:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:51 smithi123 ceph-mon[21829]: pgmap v111: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-08-19T01:36:52.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:51 smithi161 ceph-mon[21757]: pgmap v111: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-08-19T01:36:52.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.11 MB/sec execute 24 sec latency 10875.702 ms 2024-08-19T01:36:53.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 2.03 MB/sec execute 25 sec latency 11875.778 ms 2024-08-19T01:36:54.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:53 smithi123 ceph-mon[21829]: pgmap v112: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 566 KiB/s wr, 49 op/s 2024-08-19T01:36:54.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:53 smithi161 ceph-mon[21757]: pgmap v112: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 566 KiB/s wr, 49 op/s 2024-08-19T01:36:54.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:53 smithi064 ceph-mon[18515]: pgmap v112: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 566 KiB/s wr, 49 op/s 2024-08-19T01:36:54.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.95 MB/sec execute 26 sec latency 12875.874 ms 2024-08-19T01:36:55.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.88 MB/sec execute 27 sec latency 13875.962 ms 2024-08-19T01:36:56.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:55 smithi123 ceph-mon[21829]: pgmap v113: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 566 KiB/s wr, 49 op/s 2024-08-19T01:36:56.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:55 smithi161 ceph-mon[21757]: pgmap v113: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 566 KiB/s wr, 49 op/s 2024-08-19T01:36:56.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:55 smithi064 ceph-mon[18515]: pgmap v113: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 566 KiB/s wr, 49 op/s 2024-08-19T01:36:56.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.81 MB/sec execute 28 sec latency 14876.047 ms 2024-08-19T01:36:57.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.75 MB/sec execute 29 sec latency 15876.163 ms 2024-08-19T01:36:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:57 smithi123 ceph-mon[21829]: pgmap v114: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 566 KiB/s wr, 52 op/s 2024-08-19T01:36:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:57 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:36:58.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:57 smithi161 ceph-mon[21757]: pgmap v114: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 566 KiB/s wr, 52 op/s 2024-08-19T01:36:58.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:57 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:36:58.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:57 smithi064 ceph-mon[18515]: pgmap v114: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 566 KiB/s wr, 52 op/s 2024-08-19T01:36:58.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:57 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:36:58.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.69 MB/sec execute 30 sec latency 16876.248 ms 2024-08-19T01:36:59.100 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:36:58 smithi161 ceph-mon[21757]: pgmap v115: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-08-19T01:36:59.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:36:58 smithi123 ceph-mon[21829]: pgmap v115: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-08-19T01:36:59.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:36:58 smithi064 ceph-mon[18515]: pgmap v115: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-08-19T01:36:59.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.63 MB/sec execute 31 sec latency 17876.344 ms 2024-08-19T01:37:00.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.58 MB/sec execute 32 sec latency 18876.390 ms 2024-08-19T01:37:01.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:00 smithi161 ceph-mon[21757]: pgmap v116: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-08-19T01:37:01.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:00 smithi064 ceph-mon[18515]: pgmap v116: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-08-19T01:37:01.383 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.54 MB/sec execute 33 sec latency 19876.476 ms 2024-08-19T01:37:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:00 smithi123 ceph-mon[21829]: pgmap v116: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-08-19T01:37:02.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.49 MB/sec execute 34 sec latency 20876.522 ms 2024-08-19T01:37:03.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:02 smithi161 ceph-mon[21757]: pgmap v117: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-08-19T01:37:03.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:02 smithi064 ceph-mon[18515]: pgmap v117: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-08-19T01:37:03.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.45 MB/sec execute 35 sec latency 21876.615 ms 2024-08-19T01:37:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:02 smithi123 ceph-mon[21829]: pgmap v117: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-08-19T01:37:04.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.41 MB/sec execute 36 sec latency 22876.700 ms 2024-08-19T01:37:05.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:04 smithi161 ceph-mon[21757]: pgmap v118: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-08-19T01:37:05.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:04 smithi064 ceph-mon[18515]: pgmap v118: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-08-19T01:37:05.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.37 MB/sec execute 37 sec latency 23876.737 ms 2024-08-19T01:37:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:04 smithi123 ceph-mon[21829]: pgmap v118: 129 pgs: 129 active+clean; 136 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-08-19T01:37:06.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.33 MB/sec execute 38 sec latency 24876.804 ms 2024-08-19T01:37:07.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:06 smithi064 ceph-mon[18515]: pgmap v119: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 835 KiB/s wr, 45 op/s 2024-08-19T01:37:07.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.30 MB/sec execute 39 sec latency 25876.899 ms 2024-08-19T01:37:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:06 smithi123 ceph-mon[21829]: pgmap v119: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 835 KiB/s wr, 45 op/s 2024-08-19T01:37:07.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:06 smithi161 ceph-mon[21757]: pgmap v119: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 835 KiB/s wr, 45 op/s 2024-08-19T01:37:08.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.27 MB/sec execute 40 sec latency 26876.947 ms 2024-08-19T01:37:08.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:37:09.182 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:37:09.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:08 smithi064 ceph-mon[18515]: pgmap v120: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 835 KiB/s wr, 42 op/s 2024-08-19T01:37:09.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.24 MB/sec execute 41 sec latency 27877.035 ms 2024-08-19T01:37:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:08 smithi123 ceph-mon[21829]: pgmap v120: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 835 KiB/s wr, 42 op/s 2024-08-19T01:37:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:08 smithi161 ceph-mon[21757]: pgmap v120: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 835 KiB/s wr, 42 op/s 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 -- 172.21.15.64:0/1545970747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 msgr2=0x7fd16c105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 --2- 172.21.15.64:0/1545970747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c105b60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009900 tx=0x7fd15c02a410 comp rx=0 tx=0).stop 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 -- 172.21.15.64:0/1545970747 shutdown_connections 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 --2- 172.21.15.64:0/1545970747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c1018f0 0x7fd16c10d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 --2- 172.21.15.64:0/1545970747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c100f50 0x7fd16c1013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 --2- 172.21.15.64:0/1545970747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c105b60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 -- 172.21.15.64:0/1545970747 >> 172.21.15.64:0/1545970747 conn(0x7fd16c0fbc10 msgr2=0x7fd16c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 -- 172.21.15.64:0/1545970747 shutdown_connections 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.679+0000 7fd172412640 1 -- 172.21.15.64:0/1545970747 wait complete. 2024-08-19T01:37:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.680+0000 7fd172412640 1 Processor -- start 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.680+0000 7fd172412640 1 -- start start 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.680+0000 7fd172412640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c100f50 0x7fd16c0fc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.680+0000 7fd172412640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1018f0 0x7fd16c0fc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.680+0000 7fd172412640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c0fccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.680+0000 7fd172412640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c110bc0 con 0x7fd16c105780 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.680+0000 7fd172412640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c110a40 con 0x7fd16c1018f0 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd172412640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c110d40 con 0x7fd16c100f50 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c0fccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c0fccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50424/0 (socket says 172.21.15.64:50424) 2024-08-19T01:37:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 -- 172.21.15.64:0/122302130 learned_addr learned my addr 172.21.15.64:0/122302130 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd16bfff640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c100f50 0x7fd16c0fc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd16b7fe640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1018f0 0x7fd16c0fc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 -- 172.21.15.64:0/122302130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c100f50 msgr2=0x7fd16c0fc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c100f50 0x7fd16c0fc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 -- 172.21.15.64:0/122302130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1018f0 msgr2=0x7fd16c0fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1018f0 0x7fd16c0fc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 -- 172.21.15.64:0/122302130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c1a65e0 con 0x7fd16c105780 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd16bfff640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c100f50 0x7fd16c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd16b7fe640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1018f0 0x7fd16c0fc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1709af640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c0fccd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd16400b970 tx=0x7fd16400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1697fa640 1 -- 172.21.15.64:0/122302130 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164019070 con 0x7fd16c105780 2024-08-19T01:37:09.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c1a68d0 con 0x7fd16c105780 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1697fa640 1 -- 172.21.15.64:0/122302130 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd16400cb40 con 0x7fd16c105780 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd1697fa640 1 -- 172.21.15.64:0/122302130 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164007650 con 0x7fd16c105780 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.681+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c1a6e10 con 0x7fd16c105780 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.682+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c1013b0 con 0x7fd16c105780 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.684+0000 7fd1697fa640 1 -- 172.21.15.64:0/122302130 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd16401c070 con 0x7fd16c105780 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.685+0000 7fd1697fa640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd1540a2cf0 0x7fd1540a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.685+0000 7fd16bfff640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd1540a2cf0 0x7fd1540a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.685+0000 7fd1697fa640 1 -- 172.21.15.64:0/122302130 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd1640cf240 con 0x7fd16c105780 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.685+0000 7fd16bfff640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd1540a2cf0 0x7fd1540a51b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005ab0 tx=0x7fd15c005a20 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:09.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.688+0000 7fd1697fa640 1 -- 172.21.15.64:0/122302130 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd16409bbc0 con 0x7fd16c105780 2024-08-19T01:37:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.901+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd16c109cf0 con 0x7fd16c105780 2024-08-19T01:37:09.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.902+0000 7fd1697fa640 1 -- 172.21.15.64:0/122302130 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11922 (secure 0 0 0) 0x7fd16409b4c0 con 0x7fd16c105780 2024-08-19T01:37:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:37:09.903 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":28,"btime":"2024-08-19T01:36:56:739448+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:36:56.549923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:37:09.903 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 28 2024-08-19T01:37:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd1540a2cf0 msgr2=0x7fd1540a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd1540a2cf0 0x7fd1540a51b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005ab0 tx=0x7fd15c005a20 comp rx=0 tx=0).stop 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 msgr2=0x7fd16c0fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c0fccd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd16400b970 tx=0x7fd16400be30 comp rx=0 tx=0).stop 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 shutdown_connections 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd1540a2cf0 0x7fd1540a51b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd16c105780 0x7fd16c0fccd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1018f0 0x7fd16c0fc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 --2- 172.21.15.64:0/122302130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c100f50 0x7fd16c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 >> 172.21.15.64:0/122302130 conn(0x7fd16c0fbc10 msgr2=0x7fd16c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 shutdown_connections 2024-08-19T01:37:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:09.904+0000 7fd172412640 1 -- 172.21.15.64:0/122302130 wait complete. 2024-08-19T01:37:10.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:09 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/122302130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:37:10.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.21 MB/sec execute 42 sec latency 28877.109 ms 2024-08-19T01:37:10.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:09 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/122302130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:37:10.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:09 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/122302130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:37:10.850 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 4a41811a --release --await' 2024-08-19T01:37:10.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 4a41811a --release --await 2024-08-19T01:37:11.115 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:37:11.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:10 smithi064 ceph-mon[18515]: pgmap v121: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 844 KiB/s wr, 45 op/s 2024-08-19T01:37:11.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 263234 1.18 MB/sec execute 43 sec latency 29877.146 ms 2024-08-19T01:37:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:10 smithi123 ceph-mon[21829]: pgmap v121: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 844 KiB/s wr, 45 op/s 2024-08-19T01:37:11.483 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:10 smithi161 ceph-mon[21757]: pgmap v121: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 844 KiB/s wr, 45 op/s 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 -- 172.21.15.64:0/4116942817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 msgr2=0x7f32681064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 --2- 172.21.15.64:0/4116942817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681064d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3250009900 tx=0x7f325002a410 comp rx=0 tx=0).stop 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 -- 172.21.15.64:0/4116942817 shutdown_connections 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 --2- 172.21.15.64:0/4116942817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3268106c00 0x7f326810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 --2- 172.21.15.64:0/4116942817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681064d0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 --2- 172.21.15.64:0/4116942817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3268100230 0x7f3268100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 -- 172.21.15.64:0/4116942817 >> 172.21.15.64:0/4116942817 conn(0x7f32680fc2f0 msgr2=0x7f32680fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 -- 172.21.15.64:0/4116942817 shutdown_connections 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.632+0000 7f326d5dd640 1 -- 172.21.15.64:0/4116942817 wait complete. 2024-08-19T01:37:11.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.633+0000 7f326d5dd640 1 Processor -- start 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.633+0000 7f326d5dd640 1 -- start start 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.633+0000 7f326d5dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3268100230 0x7f3268198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.633+0000 7f326d5dd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.633+0000 7f326d5dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3268106c00 0x7f326819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f326d5dd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3268111c70 con 0x7f3268100be0 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f326d5dd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3268111af0 con 0x7f3268100230 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f326d5dd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3268111df0 con 0x7f3268106c00 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50438/0 (socket says 172.21.15.64:50438) 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 -- 172.21.15.64:0/441211640 learned_addr learned my addr 172.21.15.64:0/441211640 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f32677fe640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3268106c00 0x7f326819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 -- 172.21.15.64:0/441211640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3268106c00 msgr2=0x7f326819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3268106c00 0x7f326819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f3266ffd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3268100230 0x7f3268198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 -- 172.21.15.64:0/441211640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3268100230 msgr2=0x7f3268198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3268100230 0x7f3268198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 -- 172.21.15.64:0/441211640 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326819d140 con 0x7f3268100be0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f32677fe640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3268106c00 0x7f326819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f3266ffd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3268100230 0x7f3268198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f325e7fc640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681986d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3250005ab0 tx=0x7f32500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f3264ff9640 1 -- 172.21.15.64:0/441211640 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325003f030 con 0x7f3268100be0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326819d3d0 con 0x7f3268100be0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f3264ff9640 1 -- 172.21.15.64:0/441211640 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3250003ce0 con 0x7f3268100be0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f3264ff9640 1 -- 172.21.15.64:0/441211640 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250035070 con 0x7f3268100be0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.634+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32681a4d10 con 0x7f3268100be0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.636+0000 7f3264ff9640 1 -- 172.21.15.64:0/441211640 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3250042030 con 0x7f3268100be0 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 Processor -- start 2024-08-19T01:37:11.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 -- start start 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc03c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc12a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f3264ff9640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f32540a2cf0 0x7f32540a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc15370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dc1cb69c60 con 0x55dc1cc03870 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55dc1cb01b90 con 0x55dc1cc12f80 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f326f9db740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55dc1cb65c20 con 0x55dc1cc04190 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc03c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc03c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50448/0 (socket says 172.21.15.64:50448) 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f32677fe640 1 -- 172.21.15.64:0/67397959 learned_addr learned my addr 172.21.15.64:0/67397959 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.637+0000 7f325e7fc640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f32540a2cf0 0x7f32540a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f3266ffd640 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc12a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f32677fe640 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc15370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f3264ff9640 1 -- 172.21.15.64:0/441211640 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f32500f01a0 con 0x7f3268100be0 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f32677fe640 1 -- 172.21.15.64:0/67397959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 msgr2=0x55dc1cc12a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f32677fe640 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc12a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f32677fe640 1 -- 172.21.15.64:0/67397959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 msgr2=0x55dc1cc15370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f32677fe640 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc15370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f32677fe640 1 -- 172.21.15.64:0/67397959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1c879580 con 0x55dc1cc03870 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f3266ffd640 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc12a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f32677fe640 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc03c50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3260012760 tx=0x7f326000a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7803e0cd2d29e741 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f325e7fc640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f32540a2cf0 0x7f32540a51b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f32500bc010 tx=0x7f32500c0040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f325d7fa640 1 -- 172.21.15.64:0/67397959 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260018070 con 0x55dc1cc03870 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f325d7fa640 1 -- 172.21.15.64:0/67397959 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3260010d50 con 0x55dc1cc03870 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.638+0000 7f325d7fa640 1 -- 172.21.15.64:0/67397959 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32600044a0 con 0x55dc1cc03870 2024-08-19T01:37:11.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 -- 172.21.15.64:0/67397959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 msgr2=0x55dc1cc03c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc03c50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3260012760 tx=0x7f326000a880 comp rx=0 tx=0).stop 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 -- 172.21.15.64:0/67397959 shutdown_connections 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc15370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc12a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 --2- 172.21.15.64:0/67397959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc03c50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 -- 172.21.15.64:0/67397959 >> 172.21.15.64:0/67397959 conn(0x55dc1cbfff20 msgr2=0x55dc1cc00300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 -- 172.21.15.64:0/67397959 shutdown_connections 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 -- 172.21.15.64:0/67397959 wait complete. 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 -- start start 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.639+0000 7f326f9db740 1 Processor -- start 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f326f9db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc0fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f326f9db740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f326f9db740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f326f9db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55dc1cc15b50 con 0x55dc1cc03870 2024-08-19T01:37:11.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f326f9db740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55dc1cb077c0 con 0x55dc1cc12f80 2024-08-19T01:37:11.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f326f9db740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55dc1cc15cd0 con 0x55dc1cc04190 2024-08-19T01:37:11.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f326f9db740 10 client.0 _ll_register_callbacks cb 0x55dc1cb10f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:37:11.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc0fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc0fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50462/0 (socket says 172.21.15.64:50462) 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 -- 172.21.15.64:0/1508850910 learned_addr learned my addr 172.21.15.64:0/1508850910 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f3266ffd640 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 -- 172.21.15.64:0/1508850910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 msgr2=0x55dc1cc103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 -- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 msgr2=0x55dc1cc108e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc108e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 -- 172.21.15.64:0/1508850910 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1c943960 con 0x55dc1cc03870 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f32677fe640 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc0fe60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f32600123f0 tx=0x7f3260015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8510e14a9a2c9312 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.640+0000 7f324e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f324e7fc640 1 -- 172.21.15.64:0/1508850910 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260018040 con 0x55dc1cc03870 2024-08-19T01:37:11.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f324e7fc640 1 -- 172.21.15.64:0/1508850910 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3260004c40 con 0x55dc1cc03870 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3260010d50 con 0x55dc1cc03870 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f324e7fc640 1 -- 172.21.15.64:0/1508850910 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3260031050 con 0x55dc1cc03870 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 10 client.15657 fetch_fsmap learned FSMap version 28 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc1cc21bc0 con 0x55dc1cc03870 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f324e7fc640 1 -- 172.21.15.64:0/1508850910 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3260010d50 con 0x55dc1cc03870 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f324e7fc640 1 -- 172.21.15.64:0/1508850910 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f3260029020 con 0x55dc1cc03870 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 10 client.15657 fetch_fsmap finished waiting for FSMap version 28 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 10 client.15657 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 20 client.15657 populate_metadata read hostname 'smithi064' 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc22fc0 0x55dc1cc253b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 4 client.15657 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc1cb771d0 con 0x55dc1cc22fc0 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.641+0000 7f3266ffd640 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc22fc0 0x55dc1cc253b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.642+0000 7f3266ffd640 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc22fc0 0x55dc1cc253b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a27e0ee9fa14c182 server_cookie=5c5bcae62b5dff77 in_seq=0 out_seq=0 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.642+0000 7f324e7fc640 10 client.15657 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.642+0000 7f324e7fc640 1 -- 172.21.15.64:0/1508850910 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc1cb771d0 con 0x55dc1cc22fc0 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f324e7fc640 10 client.15657 handle_command_reply: tid=0 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 client.15657 shutdown 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 msgr2=0x55dc1cc0fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc0fe60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f32600123f0 tx=0x7f3260015000 comp rx=0 tx=0).stop 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 shutdown_connections 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc22fc0 0x55dc1cc253b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc12f80 0x55dc1cc108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc04190 0x55dc1cc103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f324e7fc640 0 client.15657 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 --2- 172.21.15.64:0/1508850910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc0fe60 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 >> 172.21.15.64:0/1508850910 conn(0x55dc1cbfff20 msgr2=0x55dc1cc05c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:11.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 shutdown_connections 2024-08-19T01:37:11.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 1 -- 172.21.15.64:0/1508850910 wait complete. 2024-08-19T01:37:11.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.643+0000 7f326f9db740 20 client.15657 trim_cache size 0 max 16384 2024-08-19T01:37:11.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 Processor -- start 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 -- start start 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc12190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc2aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 0x55dc1cc2af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f32681af470 con 0x55dc1cc03870 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f32681acd40 con 0x55dc1cc04190 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.660+0000 7f326f9db740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32681aab90 con 0x55dc1cc12f80 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc12190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc12190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50464/0 (socket says 172.21.15.64:50464) 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 -- 172.21.15.64:0/3041576809 learned_addr learned my addr 172.21.15.64:0/3041576809 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f3266ffd640 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc2aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 -- 172.21.15.64:0/3041576809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 msgr2=0x55dc1cc2af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 0x55dc1cc2af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 -- 172.21.15.64:0/3041576809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 msgr2=0x55dc1cc2aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc2aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 -- 172.21.15.64:0/3041576809 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1cc10e30 con 0x55dc1cc03870 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f32677fe640 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc12190 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3260012760 tx=0x7f32600107f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20342743ff2d53f server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f325cff9640 1 -- 172.21.15.64:0/3041576809 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260018040 con 0x55dc1cc03870 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f325cff9640 1 -- 172.21.15.64:0/3041576809 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3260020020 con 0x55dc1cc03870 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.661+0000 7f325cff9640 1 -- 172.21.15.64:0/3041576809 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260004000 con 0x55dc1cc03870 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 -- 172.21.15.64:0/3041576809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 msgr2=0x55dc1cc12190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc12190 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3260012760 tx=0x7f32600107f0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 -- 172.21.15.64:0/3041576809 shutdown_connections 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 0x55dc1cc2af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc2aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 --2- 172.21.15.64:0/3041576809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc12190 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 -- 172.21.15.64:0/3041576809 >> 172.21.15.64:0/3041576809 conn(0x55dc1cbfff20 msgr2=0x55dc1cc0ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 -- 172.21.15.64:0/3041576809 shutdown_connections 2024-08-19T01:37:11.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 -- 172.21.15.64:0/3041576809 wait complete. 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 -- start start 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.662+0000 7f326f9db740 1 Processor -- start 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f326f9db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc1b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f326f9db740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc16b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f326f9db740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 0x55dc1cc17130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f326f9db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f32681ab200 con 0x55dc1cc03870 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f326f9db740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f32681ac670 con 0x55dc1cc04190 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f326f9db740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32681ab8d0 con 0x55dc1cc12f80 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f326f9db740 10 client.0 _ll_register_callbacks cb 0x7f3260003eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc1b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc1b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50466/0 (socket says 172.21.15.64:50466) 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 -- 172.21.15.64:0/1027721414 learned_addr learned my addr 172.21.15.64:0/1027721414 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f3266ffd640 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc16b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 -- 172.21.15.64:0/1027721414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 msgr2=0x55dc1cc17130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 0x55dc1cc17130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 -- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 msgr2=0x55dc1cc16b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc16b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 -- 172.21.15.64:0/1027721414 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1cc17960 con 0x55dc1cc03870 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f32677fe640 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc1b7f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3260002230 tx=0x7f3260003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332652b8e5694c3e server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.663+0000 7f324e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f324e7fc640 1 -- 172.21.15.64:0/1027721414 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260018070 con 0x55dc1cc03870 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f324e7fc640 1 -- 172.21.15.64:0/1027721414 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f326001c3e0 con 0x55dc1cc03870 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3260020020 con 0x55dc1cc03870 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f324e7fc640 1 -- 172.21.15.64:0/1027721414 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f326001c620 con 0x55dc1cc03870 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 10 client.15663 fetch_fsmap learned FSMap version 28 2024-08-19T01:37:11.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc1cc8db50 con 0x55dc1cc03870 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f324e7fc640 1 -- 172.21.15.64:0/1027721414 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3260020020 con 0x55dc1cc03870 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f324e7fc640 1 -- 172.21.15.64:0/1027721414 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f3260029070 con 0x55dc1cc03870 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 10 client.15663 fetch_fsmap finished waiting for FSMap version 28 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 10 client.15663 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 20 client.15663 populate_metadata read hostname 'smithi064' 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc8e160 0x55dc1cc90580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 4 client.15663 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "4a41811a", "release": true, "await": true}] 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.664+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "4a41811a", "release": true, "await": true}) -- 0x55dc1cb716e0 con 0x55dc1cc8e160 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.665+0000 7f3266ffd640 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc8e160 0x55dc1cc90580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.665+0000 7f3266ffd640 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc8e160 0x55dc1cc90580 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4cea25be0d37dc8 server_cookie=f820152fa4218087 in_seq=0 out_seq=0 2024-08-19T01:37:11.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.665+0000 7f324e7fc640 10 client.15663 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:37:11.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f324e7fc640 1 -- 172.21.15.64:0/1027721414 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55dc1cb716e0 con 0x55dc1cc8e160 2024-08-19T01:37:11.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f324e7fc640 10 client.15663 handle_command_reply: tid=0 2024-08-19T01:37:11.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 client.15663 shutdown 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 msgr2=0x55dc1cc1b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc1b7f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3260002230 tx=0x7f3260003ba0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 shutdown_connections 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc1cc12f80 0x55dc1cc17130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55dc1cc8e160 0x55dc1cc90580 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc1cc04190 0x55dc1cc16b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 --2- 172.21.15.64:0/1027721414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dc1cc03870 0x55dc1cc1b7f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 >> 172.21.15.64:0/1027721414 conn(0x55dc1cbfff20 msgr2=0x55dc1cc0e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f324e7fc640 0 client.15663 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 shutdown_connections 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 1 -- 172.21.15.64:0/1027721414 wait complete. 2024-08-19T01:37:11.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.675+0000 7f326f9db740 20 client.15663 trim_cache size 0 max 16384 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 28, 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 16, 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "4a41811a": { 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 16, 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:37:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 150.0, 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:37:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f32540a2cf0 msgr2=0x7f32540a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f32540a2cf0 0x7f32540a51b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f32500bc010 tx=0x7f32500c0040 comp rx=0 tx=0).stop 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 msgr2=0x7f32681986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681986d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3250005ab0 tx=0x7f32500042f0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 shutdown_connections 2024-08-19T01:37:11.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f32540a2cf0 0x7f32540a51b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3268106c00 0x7f326819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268100be0 0x7f32681986d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 --2- 172.21.15.64:0/441211640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3268100230 0x7f3268198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:37:11.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 >> 172.21.15.64:0/441211640 conn(0x7f32680fc2f0 msgr2=0x7f3268105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:37:11.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 shutdown_connections 2024-08-19T01:37:11.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:37:11.676+0000 7f326d5dd640 1 -- 172.21.15.64:0/441211640 wait complete. 2024-08-19T01:37:12.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 265774 1.47 MB/sec execute 44 sec latency 30170.332 ms 2024-08-19T01:37:12.526 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '4a41811a', total seconds elapsed: 33.6 2024-08-19T01:37:13.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:12 smithi064 ceph-mon[18515]: pgmap v122: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 842 KiB/s wr, 42 op/s 2024-08-19T01:37:13.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:12 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:37:13.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 268356 1.70 MB/sec execute 45 sec latency 16.301 ms 2024-08-19T01:37:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:12 smithi123 ceph-mon[21829]: pgmap v122: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 842 KiB/s wr, 42 op/s 2024-08-19T01:37:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:12 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:37:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:12 smithi161 ceph-mon[21757]: pgmap v122: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 842 KiB/s wr, 42 op/s 2024-08-19T01:37:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:12 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:37:14.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 270526 1.94 MB/sec execute 46 sec latency 13.770 ms 2024-08-19T01:37:15.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:14 smithi064 ceph-mon[18515]: pgmap v123: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 842 KiB/s wr, 42 op/s 2024-08-19T01:37:15.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 273582 2.29 MB/sec execute 47 sec latency 12.342 ms 2024-08-19T01:37:15.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:14 smithi123 ceph-mon[21829]: pgmap v123: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 842 KiB/s wr, 42 op/s 2024-08-19T01:37:15.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:14 smithi161 ceph-mon[21757]: pgmap v123: 129 pgs: 129 active+clean; 135 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 842 KiB/s wr, 42 op/s 2024-08-19T01:37:16.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 276757 2.61 MB/sec execute 48 sec latency 20.062 ms 2024-08-19T01:37:17.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:16 smithi064 ceph-mon[18515]: pgmap v124: 129 pgs: 129 active+clean; 142 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.4 MiB/s wr, 101 op/s 2024-08-19T01:37:17.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 279255 2.75 MB/sec execute 49 sec latency 16.684 ms 2024-08-19T01:37:17.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:16 smithi123 ceph-mon[21829]: pgmap v124: 129 pgs: 129 active+clean; 142 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.4 MiB/s wr, 101 op/s 2024-08-19T01:37:17.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:16 smithi161 ceph-mon[21757]: pgmap v124: 129 pgs: 129 active+clean; 142 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.4 MiB/s wr, 101 op/s 2024-08-19T01:37:18.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 281526 2.96 MB/sec execute 50 sec latency 12.372 ms 2024-08-19T01:37:19.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:18 smithi064 ceph-mon[18515]: pgmap v125: 129 pgs: 129 active+clean; 142 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 636 KiB/s wr, 62 op/s 2024-08-19T01:37:19.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 284370 3.24 MB/sec execute 51 sec latency 15.703 ms 2024-08-19T01:37:19.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:18 smithi123 ceph-mon[21829]: pgmap v125: 129 pgs: 129 active+clean; 142 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 636 KiB/s wr, 62 op/s 2024-08-19T01:37:19.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:18 smithi161 ceph-mon[21757]: pgmap v125: 129 pgs: 129 active+clean; 142 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 636 KiB/s wr, 62 op/s 2024-08-19T01:37:20.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 287469 3.52 MB/sec execute 52 sec latency 27.078 ms 2024-08-19T01:37:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:20 smithi064 ceph-mon[18515]: pgmap v126: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 153 op/s 2024-08-19T01:37:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:20 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:37:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:20 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:37:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:20 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:37:21.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:20 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:37:21.384 INFO:tasks.workunit.client.0.smithi064.stdout: 1 289941 3.63 MB/sec execute 53 sec latency 11.699 ms 2024-08-19T01:37:21.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:20 smithi123 ceph-mon[21829]: pgmap v126: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 153 op/s 2024-08-19T01:37:21.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:20 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:37:21.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:20 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:37:21.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:20 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:37:21.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:20 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:37:21.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:20 smithi161 ceph-mon[21757]: pgmap v126: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 153 op/s 2024-08-19T01:37:21.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:20 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:37:21.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:20 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:37:21.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:20 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:37:21.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:20 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:37:22.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 292405 3.86 MB/sec execute 54 sec latency 13.017 ms 2024-08-19T01:37:23.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:22 smithi064 ceph-mon[18515]: pgmap v127: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-08-19T01:37:23.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 295112 4.05 MB/sec execute 55 sec latency 18.680 ms 2024-08-19T01:37:23.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:22 smithi123 ceph-mon[21829]: pgmap v127: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-08-19T01:37:23.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:22 smithi161 ceph-mon[21757]: pgmap v127: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-08-19T01:37:24.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 298349 4.31 MB/sec execute 56 sec latency 7.026 ms 2024-08-19T01:37:25.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:24 smithi064 ceph-mon[18515]: pgmap v128: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-08-19T01:37:25.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 301502 4.54 MB/sec execute 57 sec latency 11.368 ms 2024-08-19T01:37:25.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:24 smithi123 ceph-mon[21829]: pgmap v128: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-08-19T01:37:25.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:24 smithi161 ceph-mon[21757]: pgmap v128: 129 pgs: 129 active+clean; 151 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-08-19T01:37:26.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 303722 4.60 MB/sec execute 58 sec latency 47.623 ms 2024-08-19T01:37:27.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:26 smithi064 ceph-mon[18515]: pgmap v129: 129 pgs: 129 active+clean; 162 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-08-19T01:37:27.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 306068 4.76 MB/sec execute 59 sec latency 11.234 ms 2024-08-19T01:37:27.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:26 smithi123 ceph-mon[21829]: pgmap v129: 129 pgs: 129 active+clean; 162 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-08-19T01:37:27.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:26 smithi161 ceph-mon[21757]: pgmap v129: 129 pgs: 129 active+clean; 162 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-08-19T01:37:28.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 309173 4.98 MB/sec execute 60 sec latency 10.755 ms 2024-08-19T01:37:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:28 smithi064 ceph-mon[18515]: pgmap v130: 129 pgs: 129 active+clean; 162 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-08-19T01:37:29.250 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:28 smithi161 ceph-mon[21757]: pgmap v130: 129 pgs: 129 active+clean; 162 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-08-19T01:37:29.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 312400 5.21 MB/sec execute 61 sec latency 10.806 ms 2024-08-19T01:37:29.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:28 smithi123 ceph-mon[21829]: pgmap v130: 129 pgs: 129 active+clean; 162 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-08-19T01:37:30.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 315149 5.33 MB/sec execute 62 sec latency 18.839 ms 2024-08-19T01:37:31.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:30 smithi064 ceph-mon[18515]: pgmap v131: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-08-19T01:37:31.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 317399 5.44 MB/sec execute 63 sec latency 15.598 ms 2024-08-19T01:37:31.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:30 smithi123 ceph-mon[21829]: pgmap v131: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-08-19T01:37:31.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:30 smithi161 ceph-mon[21757]: pgmap v131: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.9 MiB/s wr, 273 op/s 2024-08-19T01:37:32.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 320056 5.58 MB/sec execute 64 sec latency 12.627 ms 2024-08-19T01:37:33.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:32 smithi064 ceph-mon[18515]: pgmap v132: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-08-19T01:37:33.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 323242 5.78 MB/sec execute 65 sec latency 9.608 ms 2024-08-19T01:37:33.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:32 smithi123 ceph-mon[21829]: pgmap v132: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-08-19T01:37:33.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:32 smithi161 ceph-mon[21757]: pgmap v132: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-08-19T01:37:34.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 326445 5.96 MB/sec execute 66 sec latency 22.446 ms 2024-08-19T01:37:35.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 329120 6.05 MB/sec execute 67 sec latency 18.988 ms 2024-08-19T01:37:35.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:34 smithi123 ceph-mon[21829]: pgmap v133: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-08-19T01:37:35.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:34 smithi161 ceph-mon[21757]: pgmap v133: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-08-19T01:37:35.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:34 smithi064 ceph-mon[18515]: pgmap v133: 129 pgs: 129 active+clean; 171 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-08-19T01:37:36.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 331394 6.16 MB/sec execute 68 sec latency 16.263 ms 2024-08-19T01:37:37.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 334151 6.28 MB/sec execute 69 sec latency 16.673 ms 2024-08-19T01:37:37.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:36 smithi123 ceph-mon[21829]: pgmap v134: 129 pgs: 129 active+clean; 181 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-08-19T01:37:37.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:36 smithi161 ceph-mon[21757]: pgmap v134: 129 pgs: 129 active+clean; 181 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-08-19T01:37:37.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:36 smithi064 ceph-mon[18515]: pgmap v134: 129 pgs: 129 active+clean; 181 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-08-19T01:37:38.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 337269 6.45 MB/sec execute 70 sec latency 8.322 ms 2024-08-19T01:37:39.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 340325 6.57 MB/sec execute 71 sec latency 14.254 ms 2024-08-19T01:37:39.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:38 smithi123 ceph-mon[21829]: pgmap v135: 129 pgs: 129 active+clean; 181 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-08-19T01:37:39.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:38 smithi161 ceph-mon[21757]: pgmap v135: 129 pgs: 129 active+clean; 181 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-08-19T01:37:39.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:38 smithi064 ceph-mon[18515]: pgmap v135: 129 pgs: 129 active+clean; 181 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-08-19T01:37:40.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 342896 6.64 MB/sec execute 72 sec latency 12.647 ms 2024-08-19T01:37:41.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 345067 6.73 MB/sec execute 73 sec latency 16.904 ms 2024-08-19T01:37:41.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:40 smithi123 ceph-mon[21829]: pgmap v136: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.8 MiB/s wr, 278 op/s 2024-08-19T01:37:41.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:40 smithi161 ceph-mon[21757]: pgmap v136: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.8 MiB/s wr, 278 op/s 2024-08-19T01:37:41.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:40 smithi064 ceph-mon[18515]: pgmap v136: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.8 MiB/s wr, 278 op/s 2024-08-19T01:37:42.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 348206 6.88 MB/sec execute 74 sec latency 10.275 ms 2024-08-19T01:37:43.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:42 smithi064 ceph-mon[18515]: pgmap v137: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-08-19T01:37:43.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 351300 7.03 MB/sec execute 75 sec latency 17.072 ms 2024-08-19T01:37:43.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:42 smithi123 ceph-mon[21829]: pgmap v137: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-08-19T01:37:43.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:42 smithi161 ceph-mon[21757]: pgmap v137: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-08-19T01:37:44.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 353766 7.06 MB/sec execute 76 sec latency 11.314 ms 2024-08-19T01:37:45.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 355877 7.14 MB/sec execute 77 sec latency 56.684 ms 2024-08-19T01:37:45.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:44 smithi123 ceph-mon[21829]: pgmap v138: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-08-19T01:37:45.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:44 smithi161 ceph-mon[21757]: pgmap v138: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-08-19T01:37:45.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:44 smithi064 ceph-mon[18515]: pgmap v138: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 186 op/s 2024-08-19T01:37:46.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 358858 7.26 MB/sec execute 78 sec latency 8.818 ms 2024-08-19T01:37:47.385 INFO:tasks.workunit.client.0.smithi064.stdout: 1 362027 7.41 MB/sec execute 79 sec latency 11.576 ms 2024-08-19T01:37:47.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:46 smithi123 ceph-mon[21829]: pgmap v139: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.1 MiB/s wr, 288 op/s 2024-08-19T01:37:47.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:46 smithi161 ceph-mon[21757]: pgmap v139: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.1 MiB/s wr, 288 op/s 2024-08-19T01:37:47.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:46 smithi064 ceph-mon[18515]: pgmap v139: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.1 MiB/s wr, 288 op/s 2024-08-19T01:37:48.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 364130 7.41 MB/sec execute 80 sec latency 67.800 ms 2024-08-19T01:37:49.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 366252 7.48 MB/sec execute 81 sec latency 16.016 ms 2024-08-19T01:37:49.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:49 smithi123 ceph-mon[21829]: pgmap v140: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 197 op/s 2024-08-19T01:37:49.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:49 smithi161 ceph-mon[21757]: pgmap v140: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 197 op/s 2024-08-19T01:37:49.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:49 smithi064 ceph-mon[18515]: pgmap v140: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 197 op/s 2024-08-19T01:37:50.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 369265 7.61 MB/sec execute 82 sec latency 30.594 ms 2024-08-19T01:37:51.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 372441 7.73 MB/sec execute 83 sec latency 12.652 ms 2024-08-19T01:37:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:51 smithi123 ceph-mon[21829]: pgmap v141: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-08-19T01:37:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:51 smithi161 ceph-mon[21757]: pgmap v141: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-08-19T01:37:51.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:51 smithi064 ceph-mon[18515]: pgmap v141: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.1 MiB/s wr, 323 op/s 2024-08-19T01:37:52.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 374929 7.75 MB/sec execute 84 sec latency 13.147 ms 2024-08-19T01:37:53.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 376755 7.80 MB/sec execute 85 sec latency 60.137 ms 2024-08-19T01:37:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:53 smithi123 ceph-mon[21829]: pgmap v142: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 227 op/s 2024-08-19T01:37:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:53 smithi161 ceph-mon[21757]: pgmap v142: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 227 op/s 2024-08-19T01:37:53.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:53 smithi064 ceph-mon[18515]: pgmap v142: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 227 op/s 2024-08-19T01:37:54.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 379829 7.92 MB/sec execute 86 sec latency 15.603 ms 2024-08-19T01:37:55.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 382583 7.98 MB/sec execute 87 sec latency 11.167 ms 2024-08-19T01:37:55.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:55 smithi123 ceph-mon[21829]: pgmap v143: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 227 op/s 2024-08-19T01:37:55.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:55 smithi161 ceph-mon[21757]: pgmap v143: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 227 op/s 2024-08-19T01:37:55.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:55 smithi064 ceph-mon[18515]: pgmap v143: 129 pgs: 129 active+clean; 207 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 227 op/s 2024-08-19T01:37:56.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 385425 8.04 MB/sec execute 88 sec latency 16.023 ms 2024-08-19T01:37:57.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 387676 8.10 MB/sec execute 89 sec latency 21.203 ms 2024-08-19T01:37:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:57 smithi123 ceph-mon[21829]: pgmap v144: 129 pgs: 129 active+clean; 215 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-08-19T01:37:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:57 smithi161 ceph-mon[21757]: pgmap v144: 129 pgs: 129 active+clean; 215 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-08-19T01:37:57.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:57 smithi064 ceph-mon[18515]: pgmap v144: 129 pgs: 129 active+clean; 215 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-08-19T01:37:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 390506 8.21 MB/sec execute 90 sec latency 100.295 ms 2024-08-19T01:37:59.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 393228 8.26 MB/sec execute 91 sec latency 13.801 ms 2024-08-19T01:37:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:37:59 smithi123 ceph-mon[21829]: pgmap v145: 129 pgs: 129 active+clean; 215 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 245 op/s 2024-08-19T01:37:59.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:37:59 smithi161 ceph-mon[21757]: pgmap v145: 129 pgs: 129 active+clean; 215 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 245 op/s 2024-08-19T01:37:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:37:59 smithi064 ceph-mon[18515]: pgmap v145: 129 pgs: 129 active+clean; 215 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 245 op/s 2024-08-19T01:38:00.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 394784 8.25 MB/sec execute 92 sec latency 62.884 ms 2024-08-19T01:38:01.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:01 smithi064 ceph-mon[18515]: pgmap v146: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 370 op/s 2024-08-19T01:38:01.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 396796 8.29 MB/sec execute 93 sec latency 56.061 ms 2024-08-19T01:38:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:01 smithi123 ceph-mon[21829]: pgmap v146: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 370 op/s 2024-08-19T01:38:01.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:01 smithi161 ceph-mon[21757]: pgmap v146: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 370 op/s 2024-08-19T01:38:02.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 399225 8.32 MB/sec execute 94 sec latency 17.230 ms 2024-08-19T01:38:03.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 401591 8.38 MB/sec execute 95 sec latency 46.921 ms 2024-08-19T01:38:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:03 smithi123 ceph-mon[21829]: pgmap v147: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-08-19T01:38:03.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:03 smithi161 ceph-mon[21757]: pgmap v147: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-08-19T01:38:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:03 smithi064 ceph-mon[18515]: pgmap v147: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-08-19T01:38:04.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 404656 8.49 MB/sec execute 96 sec latency 78.764 ms 2024-08-19T01:38:05.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 407182 8.52 MB/sec execute 97 sec latency 14.285 ms 2024-08-19T01:38:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:05 smithi123 ceph-mon[21829]: pgmap v148: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-08-19T01:38:05.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:05 smithi161 ceph-mon[21757]: pgmap v148: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-08-19T01:38:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:05 smithi064 ceph-mon[18515]: pgmap v148: 129 pgs: 129 active+clean; 223 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-08-19T01:38:06.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 409508 8.57 MB/sec execute 98 sec latency 17.274 ms 2024-08-19T01:38:07.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 412272 8.63 MB/sec execute 99 sec latency 19.123 ms 2024-08-19T01:38:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:07 smithi123 ceph-mon[21829]: pgmap v149: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.7 MiB/s wr, 390 op/s 2024-08-19T01:38:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:07 smithi161 ceph-mon[21757]: pgmap v149: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.7 MiB/s wr, 390 op/s 2024-08-19T01:38:07.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:07 smithi064 ceph-mon[18515]: pgmap v149: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.7 MiB/s wr, 390 op/s 2024-08-19T01:38:08.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 415407 8.73 MB/sec execute 100 sec latency 12.932 ms 2024-08-19T01:38:09.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 417835 8.75 MB/sec execute 101 sec latency 52.043 ms 2024-08-19T01:38:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:09 smithi123 ceph-mon[21829]: pgmap v150: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-08-19T01:38:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:09 smithi161 ceph-mon[21757]: pgmap v150: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-08-19T01:38:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:09 smithi064 ceph-mon[18515]: pgmap v150: 129 pgs: 129 active+clean; 230 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 271 op/s 2024-08-19T01:38:10.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 419835 8.76 MB/sec execute 102 sec latency 14.013 ms 2024-08-19T01:38:11.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 422751 8.84 MB/sec execute 103 sec latency 8.802 ms 2024-08-19T01:38:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:11 smithi123 ceph-mon[21829]: pgmap v151: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 394 op/s 2024-08-19T01:38:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:11 smithi161 ceph-mon[21757]: pgmap v151: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 394 op/s 2024-08-19T01:38:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:11 smithi064 ceph-mon[18515]: pgmap v151: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 394 op/s 2024-08-19T01:38:12.386 INFO:tasks.workunit.client.0.smithi064.stdout: 1 425925 8.94 MB/sec execute 104 sec latency 26.979 ms 2024-08-19T01:38:13.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 428684 8.98 MB/sec execute 105 sec latency 11.253 ms 2024-08-19T01:38:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:13 smithi123 ceph-mon[21829]: pgmap v152: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-08-19T01:38:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:13 smithi161 ceph-mon[21757]: pgmap v152: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-08-19T01:38:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:13 smithi064 ceph-mon[18515]: pgmap v152: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-08-19T01:38:14.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 430330 8.97 MB/sec execute 106 sec latency 108.427 ms 2024-08-19T01:38:15.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 433109 9.05 MB/sec execute 107 sec latency 76.731 ms 2024-08-19T01:38:15.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:15 smithi123 ceph-mon[21829]: pgmap v153: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-08-19T01:38:15.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:15 smithi161 ceph-mon[21757]: pgmap v153: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-08-19T01:38:15.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:15 smithi064 ceph-mon[18515]: pgmap v153: 129 pgs: 129 active+clean; 239 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-08-19T01:38:16.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 436286 9.13 MB/sec execute 108 sec latency 11.882 ms 2024-08-19T01:38:17.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 438743 9.12 MB/sec execute 109 sec latency 15.741 ms 2024-08-19T01:38:17.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:17 smithi123 ceph-mon[21829]: pgmap v154: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 392 op/s 2024-08-19T01:38:17.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:17 smithi161 ceph-mon[21757]: pgmap v154: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 392 op/s 2024-08-19T01:38:17.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:17 smithi064 ceph-mon[18515]: pgmap v154: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 392 op/s 2024-08-19T01:38:18.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 440928 9.17 MB/sec execute 110 sec latency 17.564 ms 2024-08-19T01:38:19.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 443792 9.24 MB/sec execute 111 sec latency 56.993 ms 2024-08-19T01:38:19.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:19 smithi123 ceph-mon[21829]: pgmap v155: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 246 op/s 2024-08-19T01:38:19.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:19 smithi161 ceph-mon[21757]: pgmap v155: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 246 op/s 2024-08-19T01:38:19.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:19 smithi064 ceph-mon[18515]: pgmap v155: 129 pgs: 129 active+clean; 249 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 246 op/s 2024-08-19T01:38:20.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 445840 9.22 MB/sec execute 112 sec latency 15.896 ms 2024-08-19T01:38:21.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 447890 9.25 MB/sec execute 113 sec latency 14.269 ms 2024-08-19T01:38:21.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:21 smithi123 ceph-mon[21829]: pgmap v156: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 368 op/s 2024-08-19T01:38:21.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:21 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:38:21.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:21 smithi161 ceph-mon[21757]: pgmap v156: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 368 op/s 2024-08-19T01:38:21.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:21 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:38:21.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:21 smithi064 ceph-mon[18515]: pgmap v156: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 368 op/s 2024-08-19T01:38:21.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:21 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:38:22.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 450632 9.32 MB/sec execute 114 sec latency 14.348 ms 2024-08-19T01:38:22.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:38:22.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:38:22.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:22 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:38:22.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:38:22.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:38:22.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:22 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:38:22.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:38:22.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:38:22.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:22 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:38:23.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 452636 9.30 MB/sec execute 115 sec latency 14.334 ms 2024-08-19T01:38:23.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:23 smithi123 ceph-mon[21829]: pgmap v157: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-08-19T01:38:23.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:23 smithi161 ceph-mon[21757]: pgmap v157: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-08-19T01:38:23.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:23 smithi064 ceph-mon[18515]: pgmap v157: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-08-19T01:38:24.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 454699 9.34 MB/sec execute 116 sec latency 86.408 ms 2024-08-19T01:38:25.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 457024 9.36 MB/sec execute 117 sec latency 16.586 ms 2024-08-19T01:38:25.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:25 smithi123 ceph-mon[21829]: pgmap v158: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-08-19T01:38:25.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:25 smithi161 ceph-mon[21757]: pgmap v158: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-08-19T01:38:25.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:25 smithi064 ceph-mon[18515]: pgmap v158: 129 pgs: 129 active+clean; 258 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-08-19T01:38:26.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 458693 9.45 MB/sec execute 118 sec latency 75.060 ms 2024-08-19T01:38:27.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 460927 9.51 MB/sec execute 119 sec latency 20.325 ms 2024-08-19T01:38:27.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:27 smithi123 ceph-mon[21829]: pgmap v159: 129 pgs: 129 active+clean; 265 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.7 MiB/s wr, 352 op/s 2024-08-19T01:38:27.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:27 smithi161 ceph-mon[21757]: pgmap v159: 129 pgs: 129 active+clean; 265 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.7 MiB/s wr, 352 op/s 2024-08-19T01:38:27.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:27 smithi064 ceph-mon[18515]: pgmap v159: 129 pgs: 129 active+clean; 265 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.7 MiB/s wr, 352 op/s 2024-08-19T01:38:28.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 463396 9.56 MB/sec execute 120 sec latency 15.351 ms 2024-08-19T01:38:29.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 465906 9.60 MB/sec execute 121 sec latency 47.540 ms 2024-08-19T01:38:29.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:29 smithi123 ceph-mon[21829]: pgmap v160: 129 pgs: 129 active+clean; 265 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 229 op/s 2024-08-19T01:38:29.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:29 smithi161 ceph-mon[21757]: pgmap v160: 129 pgs: 129 active+clean; 265 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 229 op/s 2024-08-19T01:38:29.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:29 smithi064 ceph-mon[18515]: pgmap v160: 129 pgs: 129 active+clean; 265 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 229 op/s 2024-08-19T01:38:30.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 468346 9.62 MB/sec execute 122 sec latency 46.645 ms 2024-08-19T01:38:31.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 470685 9.64 MB/sec execute 123 sec latency 15.367 ms 2024-08-19T01:38:31.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:31 smithi123 ceph-mon[21829]: pgmap v161: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:38:31.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:31 smithi161 ceph-mon[21757]: pgmap v161: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:38:31.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:31 smithi064 ceph-mon[18515]: pgmap v161: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:38:32.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 473236 9.68 MB/sec execute 124 sec latency 18.215 ms 2024-08-19T01:38:33.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 476081 9.74 MB/sec execute 125 sec latency 14.334 ms 2024-08-19T01:38:33.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:33 smithi123 ceph-mon[21829]: pgmap v162: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-08-19T01:38:33.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:33 smithi161 ceph-mon[21757]: pgmap v162: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-08-19T01:38:33.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:33 smithi064 ceph-mon[18515]: pgmap v162: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-08-19T01:38:34.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 477737 9.72 MB/sec execute 126 sec latency 51.224 ms 2024-08-19T01:38:35.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 480141 9.75 MB/sec execute 127 sec latency 48.719 ms 2024-08-19T01:38:35.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:35 smithi123 ceph-mon[21829]: pgmap v163: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-08-19T01:38:35.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:35 smithi161 ceph-mon[21757]: pgmap v163: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-08-19T01:38:35.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:35 smithi064 ceph-mon[18515]: pgmap v163: 129 pgs: 129 active+clean; 273 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-08-19T01:38:36.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 482857 9.78 MB/sec execute 128 sec latency 12.654 ms 2024-08-19T01:38:37.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 484878 9.79 MB/sec execute 129 sec latency 40.526 ms 2024-08-19T01:38:37.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:37 smithi161 ceph-mon[21757]: pgmap v164: 129 pgs: 129 active+clean; 281 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 320 op/s 2024-08-19T01:38:37.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:37 smithi064 ceph-mon[18515]: pgmap v164: 129 pgs: 129 active+clean; 281 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 320 op/s 2024-08-19T01:38:37.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:37 smithi123 ceph-mon[21829]: pgmap v164: 129 pgs: 129 active+clean; 281 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 320 op/s 2024-08-19T01:38:38.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 487380 9.82 MB/sec execute 130 sec latency 35.092 ms 2024-08-19T01:38:39.387 INFO:tasks.workunit.client.0.smithi064.stdout: 1 489638 9.83 MB/sec execute 131 sec latency 60.574 ms 2024-08-19T01:38:39.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:39 smithi161 ceph-mon[21757]: pgmap v165: 129 pgs: 129 active+clean; 281 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 212 op/s 2024-08-19T01:38:39.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:39 smithi064 ceph-mon[18515]: pgmap v165: 129 pgs: 129 active+clean; 281 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 212 op/s 2024-08-19T01:38:39.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:39 smithi123 ceph-mon[21829]: pgmap v165: 129 pgs: 129 active+clean; 281 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 212 op/s 2024-08-19T01:38:40.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 491718 9.85 MB/sec execute 132 sec latency 26.394 ms 2024-08-19T01:38:41.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 494392 9.88 MB/sec execute 133 sec latency 14.755 ms 2024-08-19T01:38:41.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:41 smithi161 ceph-mon[21757]: pgmap v166: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 315 op/s 2024-08-19T01:38:41.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:41 smithi064 ceph-mon[18515]: pgmap v166: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 315 op/s 2024-08-19T01:38:41.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:41 smithi123 ceph-mon[21829]: pgmap v166: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 315 op/s 2024-08-19T01:38:42.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 497213 9.92 MB/sec execute 134 sec latency 18.153 ms 2024-08-19T01:38:43.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 499383 9.92 MB/sec execute 135 sec latency 12.702 ms 2024-08-19T01:38:43.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:43 smithi161 ceph-mon[21757]: pgmap v167: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-08-19T01:38:43.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:43 smithi064 ceph-mon[18515]: pgmap v167: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-08-19T01:38:43.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:43 smithi123 ceph-mon[21829]: pgmap v167: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-08-19T01:38:44.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 501608 9.95 MB/sec execute 136 sec latency 113.352 ms 2024-08-19T01:38:45.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 504135 9.97 MB/sec execute 137 sec latency 31.932 ms 2024-08-19T01:38:45.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:45 smithi161 ceph-mon[21757]: pgmap v168: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-08-19T01:38:45.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:45 smithi064 ceph-mon[18515]: pgmap v168: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-08-19T01:38:45.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:45 smithi123 ceph-mon[21829]: pgmap v168: 129 pgs: 129 active+clean; 288 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-08-19T01:38:46.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 506317 9.98 MB/sec execute 138 sec latency 12.416 ms 2024-08-19T01:38:47.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 508838 10.01 MB/sec execute 139 sec latency 16.406 ms 2024-08-19T01:38:47.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:47 smithi161 ceph-mon[21757]: pgmap v169: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 311 op/s 2024-08-19T01:38:47.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:47 smithi064 ceph-mon[18515]: pgmap v169: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 311 op/s 2024-08-19T01:38:47.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:47 smithi123 ceph-mon[21829]: pgmap v169: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 311 op/s 2024-08-19T01:38:48.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 511834 10.07 MB/sec execute 140 sec latency 14.018 ms 2024-08-19T01:38:49.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 513927 10.05 MB/sec execute 141 sec latency 34.495 ms 2024-08-19T01:38:49.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:49 smithi161 ceph-mon[21757]: pgmap v170: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 208 op/s 2024-08-19T01:38:49.467 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:49 smithi123 ceph-mon[21829]: pgmap v170: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 208 op/s 2024-08-19T01:38:49.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:49 smithi064 ceph-mon[18515]: pgmap v170: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 208 op/s 2024-08-19T01:38:50.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 515864 10.07 MB/sec execute 142 sec latency 96.832 ms 2024-08-19T01:38:51.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 518632 10.12 MB/sec execute 143 sec latency 19.837 ms 2024-08-19T01:38:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:51 smithi161 ceph-mon[21757]: pgmap v171: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 302 op/s 2024-08-19T01:38:51.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:51 smithi064 ceph-mon[18515]: pgmap v171: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 302 op/s 2024-08-19T01:38:51.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:51 smithi123 ceph-mon[21829]: pgmap v171: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 302 op/s 2024-08-19T01:38:52.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 520656 10.10 MB/sec execute 144 sec latency 17.634 ms 2024-08-19T01:38:53.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 522897 10.12 MB/sec execute 145 sec latency 38.128 ms 2024-08-19T01:38:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:53 smithi161 ceph-mon[21757]: pgmap v172: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:38:53.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:53 smithi064 ceph-mon[18515]: pgmap v172: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:38:53.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:53 smithi123 ceph-mon[21829]: pgmap v172: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:38:54.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 525563 10.15 MB/sec execute 146 sec latency 24.314 ms 2024-08-19T01:38:55.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 527294 10.15 MB/sec execute 147 sec latency 66.792 ms 2024-08-19T01:38:55.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:55 smithi161 ceph-mon[21757]: pgmap v173: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:38:55.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:55 smithi064 ceph-mon[18515]: pgmap v173: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:38:55.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:55 smithi123 ceph-mon[21829]: pgmap v173: 129 pgs: 129 active+clean; 314 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:38:56.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 529808 10.17 MB/sec execute 148 sec latency 42.127 ms 2024-08-19T01:38:57.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 532702 10.20 MB/sec execute 149 sec latency 13.559 ms 2024-08-19T01:38:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:57 smithi161 ceph-mon[21757]: pgmap v174: 129 pgs: 129 active+clean; 320 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 297 op/s 2024-08-19T01:38:57.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:57 smithi064 ceph-mon[18515]: pgmap v174: 129 pgs: 129 active+clean; 320 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 297 op/s 2024-08-19T01:38:57.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:57 smithi123 ceph-mon[21829]: pgmap v174: 129 pgs: 129 active+clean; 320 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 297 op/s 2024-08-19T01:38:58.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 534713 10.20 MB/sec execute 150 sec latency 13.944 ms 2024-08-19T01:38:59.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 537168 10.23 MB/sec execute 151 sec latency 23.414 ms 2024-08-19T01:38:59.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:38:59 smithi161 ceph-mon[21757]: pgmap v175: 129 pgs: 129 active+clean; 320 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 192 op/s 2024-08-19T01:38:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:38:59 smithi064 ceph-mon[18515]: pgmap v175: 129 pgs: 129 active+clean; 320 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 192 op/s 2024-08-19T01:38:59.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:38:59 smithi123 ceph-mon[21829]: pgmap v175: 129 pgs: 129 active+clean; 320 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 192 op/s 2024-08-19T01:39:00.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 539303 10.23 MB/sec execute 152 sec latency 82.384 ms 2024-08-19T01:39:01.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 539728 10.19 MB/sec execute 153 sec latency 151.327 ms 2024-08-19T01:39:01.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:01 smithi161 ceph-mon[21757]: pgmap v176: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.7 MiB/s wr, 295 op/s 2024-08-19T01:39:01.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:01 smithi064 ceph-mon[18515]: pgmap v176: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.7 MiB/s wr, 295 op/s 2024-08-19T01:39:01.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:01 smithi123 ceph-mon[21829]: pgmap v176: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.7 MiB/s wr, 295 op/s 2024-08-19T01:39:02.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 541638 10.18 MB/sec execute 154 sec latency 14.284 ms 2024-08-19T01:39:03.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 544095 10.21 MB/sec execute 155 sec latency 15.416 ms 2024-08-19T01:39:03.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:03 smithi161 ceph-mon[21757]: pgmap v177: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-08-19T01:39:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:03 smithi064 ceph-mon[18515]: pgmap v177: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-08-19T01:39:03.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:03 smithi123 ceph-mon[21829]: pgmap v177: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-08-19T01:39:03.955 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-19T01:39:03.955 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.2 2024-08-19T01:39:03.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:39:04.226 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:04.388 INFO:tasks.workunit.client.0.smithi064.stdout: 1 546543 10.22 MB/sec execute 156 sec latency 20.377 ms 2024-08-19T01:39:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.789+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/2723624305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 msgr2=0x7fcbd41026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.789+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/2723624305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 0x7fcbd41026e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009900 tx=0x7fcbc402a480 comp rx=0 tx=0).stop 2024-08-19T01:39:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/2723624305 shutdown_connections 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/2723624305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbd4102c20 0x7fcbd410d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/2723624305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 0x7fcbd41026e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/2723624305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 0x7fcbd4100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/2723624305 >> 172.21.15.64:0/2723624305 conn(0x7fcbd40fc500 msgr2=0x7fcbd40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/2723624305 shutdown_connections 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/2723624305 wait complete. 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 Processor -- start 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.790+0000 7fcbdc0fd640 1 -- start start 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbdc0fd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 0x7fcbd4196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbdc0fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 0x7fcbd41973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbdc0fd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbd4102c20 0x7fcbd4197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:04.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbdc0fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd4110d00 con 0x7fcbd41022a0 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbdc0fd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd4110b80 con 0x7fcbd4102c20 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbdc0fd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd4110e80 con 0x7fcbd4100b50 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9e99640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 0x7fcbd4196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9698640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 0x7fcbd41973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9e99640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 0x7fcbd4196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:47816/0 (socket says 172.21.15.64:47816) 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9e99640 1 -- 172.21.15.64:0/538873409 learned_addr learned my addr 172.21.15.64:0/538873409 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbda69a640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbd4102c20 0x7fcbd4197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9698640 1 -- 172.21.15.64:0/538873409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 msgr2=0x7fcbd4196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9698640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 0x7fcbd4196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9698640 1 -- 172.21.15.64:0/538873409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbd4102c20 msgr2=0x7fcbd4197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9698640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbd4102c20 0x7fcbd4197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9698640 1 -- 172.21.15.64:0/538873409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd41a6710 con 0x7fcbd41022a0 2024-08-19T01:39:04.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9e99640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 0x7fcbd4196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:04.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.791+0000 7fcbd9698640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 0x7fcbd41973e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcbc402a9a0 tx=0x7fcbc4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.792+0000 7fcbcaffd640 1 -- 172.21.15.64:0/538873409 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc403f070 con 0x7fcbd41022a0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.792+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd41a69a0 con 0x7fcbd41022a0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.792+0000 7fcbcaffd640 1 -- 172.21.15.64:0/538873409 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbc4031c20 con 0x7fcbd41022a0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.792+0000 7fcbcaffd640 1 -- 172.21.15.64:0/538873409 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4035030 con 0x7fcbd41022a0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.792+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd41a6ed0 con 0x7fcbd41022a0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.793+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd4100f30 con 0x7fcbd41022a0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.794+0000 7fcbcaffd640 1 -- 172.21.15.64:0/538873409 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcbc4042070 con 0x7fcbd41022a0 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.794+0000 7fcbcaffd640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcbc00a2cf0 0x7fcbc00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:04.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.795+0000 7fcbd9e99640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcbc00a2cf0 0x7fcbc00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:04.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.795+0000 7fcbcaffd640 1 -- 172.21.15.64:0/538873409 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcbc40efb40 con 0x7fcbd41022a0 2024-08-19T01:39:04.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.795+0000 7fcbd9e99640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcbc00a2cf0 0x7fcbc00a51b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0011500 tx=0x7fcbd0005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:04.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:04.799+0000 7fcbcaffd640 1 -- 172.21.15.64:0/538873409 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fcbc40bbd60 con 0x7fcbd41022a0 2024-08-19T01:39:05.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.026+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbd4108dc0 con 0x7fcbd41022a0 2024-08-19T01:39:05.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.027+0000 7fcbcaffd640 1 -- 172.21.15.64:0/538873409 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7fcbc40c0060 con 0x7fcbd41022a0 2024-08-19T01:39:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:39:05.028 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:39:05.029 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:39:05.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcbc00a2cf0 msgr2=0x7fcbc00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:05.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcbc00a2cf0 0x7fcbc00a51b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0011500 tx=0x7fcbd0005f90 comp rx=0 tx=0).stop 2024-08-19T01:39:05.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 msgr2=0x7fcbd41973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 0x7fcbd41973e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcbc402a9a0 tx=0x7fcbc4003b90 comp rx=0 tx=0).stop 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 shutdown_connections 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcbc00a2cf0 0x7fcbc00a51b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbd4102c20 0x7fcbd4197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbd41022a0 0x7fcbd41973e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 --2- 172.21.15.64:0/538873409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd4100b50 0x7fcbd4196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 >> 172.21.15.64:0/538873409 conn(0x7fcbd40fc500 msgr2=0x7fcbd40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 shutdown_connections 2024-08-19T01:39:05.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:05.029+0000 7fcbdc0fd640 1 -- 172.21.15.64:0/538873409 wait complete. 2024-08-19T01:39:05.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 548505 10.23 MB/sec execute 157 sec latency 45.787 ms 2024-08-19T01:39:05.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:05 smithi161 ceph-mon[21757]: pgmap v178: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-08-19T01:39:05.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:05 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/538873409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:05 smithi064 ceph-mon[18515]: pgmap v178: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-08-19T01:39:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:05 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/538873409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:05.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:05 smithi123 ceph-mon[21829]: pgmap v178: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-08-19T01:39:05.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:05 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/538873409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:06.102 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 139.2 --await-for 45.0' 2024-08-19T01:39:06.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 139.2 --await-for 45.0 2024-08-19T01:39:06.366 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:06.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 550879 10.25 MB/sec execute 158 sec latency 40.604 ms 2024-08-19T01:39:06.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 -- 172.21.15.64:0/131027354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff160102c20 msgr2=0x7ff16010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 --2- 172.21.15.64:0/131027354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff160102c20 0x7ff16010d770 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff158009ca0 tx=0x7ff15802a950 comp rx=0 tx=0).stop 2024-08-19T01:39:06.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 -- 172.21.15.64:0/131027354 shutdown_connections 2024-08-19T01:39:06.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 --2- 172.21.15.64:0/131027354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff160102c20 0x7ff16010d770 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 --2- 172.21.15.64:0/131027354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601022a0 0x7ff1601026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 --2- 172.21.15.64:0/131027354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160100b50 0x7ff160100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 -- 172.21.15.64:0/131027354 >> 172.21.15.64:0/131027354 conn(0x7ff1600fc500 msgr2=0x7ff1600fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 -- 172.21.15.64:0/131027354 shutdown_connections 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.890+0000 7ff166a3a640 1 -- 172.21.15.64:0/131027354 wait complete. 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.891+0000 7ff166a3a640 1 Processor -- start 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.891+0000 7ff166a3a640 1 -- start start 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.891+0000 7ff166a3a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160100b50 0x7ff1601990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.891+0000 7ff166a3a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1601022a0 0x7ff1601995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff166a3a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff160102c20 0x7ff1601a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff166a3a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff160110cb0 con 0x7ff1601022a0 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff166a3a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff160110b30 con 0x7ff160100b50 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff166a3a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff160110e30 con 0x7ff160102c20 2024-08-19T01:39:06.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1601022a0 0x7ff1601995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1601022a0 0x7ff1601995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49366/0 (socket says 172.21.15.64:49366) 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 -- 172.21.15.64:0/2055710976 learned_addr learned my addr 172.21.15.64:0/2055710976 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160100b50 0x7ff1601990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff164fd7640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff160102c20 0x7ff1601a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 -- 172.21.15.64:0/2055710976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff160102c20 msgr2=0x7ff1601a4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff160102c20 0x7ff1601a4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 -- 172.21.15.64:0/2055710976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160100b50 msgr2=0x7ff1601990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160100b50 0x7ff1601990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 -- 172.21.15.64:0/2055710976 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1601a4660 con 0x7ff1601022a0 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15ffff640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160100b50 0x7ff1601990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1601022a0 0x7ff1601995e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff15400e9f0 tx=0x7ff15400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15d7fa640 1 -- 172.21.15.64:0/2055710976 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15400cd80 con 0x7ff1601022a0 2024-08-19T01:39:06.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1601a4830 con 0x7ff1601022a0 2024-08-19T01:39:06.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15d7fa640 1 -- 172.21.15.64:0/2055710976 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff154005500 con 0x7ff1601022a0 2024-08-19T01:39:06.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff15d7fa640 1 -- 172.21.15.64:0/2055710976 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154010690 con 0x7ff1601022a0 2024-08-19T01:39:06.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.892+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1601a4cf0 con 0x7ff1601022a0 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff15d7fa640 1 -- 172.21.15.64:0/2055710976 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff1540046c0 con 0x7ff1601022a0 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 Processor -- start 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 -- start start 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff15d7fa640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff1480a2cf0 0x7ff1480a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff164fd7640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff1480a2cf0 0x7ff1480a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfee130 0x555f2bfee510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfeea50 0x555f2bff8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff15d7fa640 1 -- 172.21.15.64:0/2055710976 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff1540cfba0 con 0x7ff1601022a0 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bffb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555f2bf53a70 con 0x555f2bfee130 2024-08-19T01:39:06.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x555f2beeb780 con 0x555f2bff9310 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.895+0000 7ff168e38740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555f2bf4fa30 con 0x555f2bfeea50 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfee130 0x555f2bfee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfeea50 0x555f2bff8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfeea50 0x555f2bff8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:47842/0 (socket says 172.21.15.64:47842) 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfee130 0x555f2bfee510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49378/0 (socket says 172.21.15.64:49378) 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1203738816 learned_addr learned my addr 172.21.15.64:0/1203738816 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff164fd7640 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bffb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1203738816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 msgr2=0x555f2bffb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bffb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1203738816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfee130 msgr2=0x555f2bfee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfee130 0x555f2bfee510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1203738816 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2bcc5b90 con 0x555f2bfeea50 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff164fd7640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff1480a2cf0 0x7ff1480a51b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x555f2bfeba40 tx=0x7ff158005ef0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfeea50 0x555f2bff8dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1540a4450 tx=0x7ff15409b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96a9d60ff5a476b6 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff1467fc640 1 -- 172.21.15.64:0/1203738816 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1540a00c0 con 0x555f2bfeea50 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff1467fc640 1 -- 172.21.15.64:0/1203738816 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1540a4980 con 0x555f2bfeea50 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.896+0000 7ff1467fc640 1 -- 172.21.15.64:0/1203738816 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15409b160 con 0x555f2bfeea50 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 -- 172.21.15.64:0/1203738816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfeea50 msgr2=0x555f2bff8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfeea50 0x555f2bff8dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1540a4450 tx=0x7ff15409b670 comp rx=0 tx=0).stop 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 -- 172.21.15.64:0/1203738816 shutdown_connections 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bffb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfeea50 0x555f2bff8dd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 --2- 172.21.15.64:0/1203738816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfee130 0x555f2bfee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 -- 172.21.15.64:0/1203738816 >> 172.21.15.64:0/1203738816 conn(0x555f2bfea6c0 msgr2=0x555f2bfeaaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 -- 172.21.15.64:0/1203738816 shutdown_connections 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.897+0000 7ff168e38740 1 -- 172.21.15.64:0/1203738816 wait complete. 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 -- start start 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 Processor -- start 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2bff66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bff6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555f2bffbee0 con 0x555f2bfeea50 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x555f2bf0e990 con 0x555f2bff9310 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555f2bffc060 con 0x555f2bfee130 2024-08-19T01:39:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff168e38740 10 client.0 _ll_register_callbacks cb 0x555f2bc63090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2bff66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff164fd7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bff6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:47854/0 (socket says 172.21.15.64:47854) 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff164fd7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bff6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:43072/0 (socket says 172.21.15.64:43072) 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.898+0000 7ff15ffff640 1 -- 172.21.15.64:0/2462610772 learned_addr learned my addr 172.21.15.64:0/2462610772 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff15ffff640 1 -- 172.21.15.64:0/2462610772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 msgr2=0x555f2bff6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff15ffff640 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bff6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff15ffff640 1 -- 172.21.15.64:0/2462610772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 msgr2=0x555f2bff66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff15ffff640 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2bff66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff15ffff640 1 -- 172.21.15.64:0/2462610772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f2bf0d6c0 con 0x555f2bfee130 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2bff66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff15ffff640 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff6160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff150012d20 tx=0x7ff15000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60db38fc2d891870 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:06.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff13f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff13f7fe640 1 -- 172.21.15.64:0/2462610772 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15000fe20 con 0x555f2bfee130 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1540a4980 con 0x555f2bfee130 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff13f7fe640 1 -- 172.21.15.64:0/2462610772 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff150010430 con 0x555f2bfee130 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff13f7fe640 1 -- 172.21.15.64:0/2462610772 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff150010630 con 0x555f2bfee130 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff168e38740 10 client.25129 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.899+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f2c007ba0 con 0x555f2bfee130 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff13f7fe640 1 -- 172.21.15.64:0/2462610772 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1540a4980 con 0x555f2bfee130 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff13f7fe640 1 -- 172.21.15.64:0/2462610772 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ff15001c940 con 0x555f2bfee130 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff168e38740 10 client.25129 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff168e38740 10 client.25129 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff168e38740 20 client.25129 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff168e38740 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c008fb0 0x555f2c00b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff168e38740 4 client.25129 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:39:06.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f2bf611a0 con 0x555f2c008fb0 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c008fb0 0x555f2c00b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c008fb0 0x555f2c00b450 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c6e12868b4e9a61 server_cookie=562c892e3313d1ef in_seq=0 out_seq=0 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.900+0000 7ff13f7fe640 10 client.25129 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff13f7fe640 1 -- 172.21.15.64:0/2462610772 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555f2bf611a0 con 0x555f2c008fb0 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff13f7fe640 10 client.25129 handle_command_reply: tid=0 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 client.25129 shutdown 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 msgr2=0x555f2bff6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff6160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff150012d20 tx=0x7ff15000f790 comp rx=0 tx=0).stop 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 shutdown_connections 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c008fb0 0x555f2c00b450 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2bff6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2bff66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 --2- 172.21.15.64:0/2462610772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff6160 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff13f7fe640 0 client.25129 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 >> 172.21.15.64:0/2462610772 conn(0x555f2bfea6c0 msgr2=0x555f2bff03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:06.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 shutdown_connections 2024-08-19T01:39:06.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 1 -- 172.21.15.64:0/2462610772 wait complete. 2024-08-19T01:39:06.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.901+0000 7ff168e38740 20 client.25129 trim_cache size 0 max 16384 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 Processor -- start 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 -- start start 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2c010d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2c0112b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff1601b1750 con 0x555f2bfeea50 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff1601af020 con 0x555f2bff9310 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff168e38740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff1601ace70 con 0x555f2bfee130 2024-08-19T01:39:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2c010d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2c010d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49406/0 (socket says 172.21.15.64:49406) 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1039252557 learned_addr learned my addr 172.21.15.64:0/1039252557 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15ffff640 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff164fd7640 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2c0112b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1039252557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 msgr2=0x555f2bff50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1039252557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 msgr2=0x555f2c0112b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2c0112b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff15f7fe640 1 -- 172.21.15.64:0/1039252557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2bfee880 con 0x555f2bfeea50 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.920+0000 7ff164fd7640 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2c0112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.921+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2c010d70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff154098170 tx=0x7ff154004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db447460695a76cc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.921+0000 7ff145ffb640 1 -- 172.21.15.64:0/1039252557 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1540a4980 con 0x555f2bfeea50 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.921+0000 7ff145ffb640 1 -- 172.21.15.64:0/1039252557 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1540a4bd0 con 0x555f2bfeea50 2024-08-19T01:39:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.921+0000 7ff145ffb640 1 -- 172.21.15.64:0/1039252557 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15409b3c0 con 0x555f2bfeea50 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.921+0000 7ff168e38740 1 -- 172.21.15.64:0/1039252557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 msgr2=0x555f2c010d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.921+0000 7ff168e38740 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2c010d70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff154098170 tx=0x7ff154004b60 comp rx=0 tx=0).stop 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 -- 172.21.15.64:0/1039252557 shutdown_connections 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bff9310 0x555f2c0112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bfeea50 0x555f2c010d70 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 --2- 172.21.15.64:0/1039252557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2bff50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 -- 172.21.15.64:0/1039252557 >> 172.21.15.64:0/1039252557 conn(0x555f2bfea6c0 msgr2=0x555f2bff81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 -- 172.21.15.64:0/1039252557 shutdown_connections 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 -- 172.21.15.64:0/1039252557 wait complete. 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 -- start start 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.922+0000 7ff168e38740 1 Processor -- start 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2c001a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bfeea50 0x555f2bff2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bff9310 0x555f2bff3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff1601ad4e0 con 0x555f2bff9310 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff1601ae950 con 0x555f2bfeea50 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff1601adbb0 con 0x555f2bfee130 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 10 client.0 _ll_register_callbacks cb 0x7ff150009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff164fd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bff9310 0x555f2bff3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2c001a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bfeea50 0x555f2bff2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff164fd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bff9310 0x555f2bff3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49412/0 (socket says 172.21.15.64:49412) 2024-08-19T01:39:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff164fd7640 1 -- 172.21.15.64:0/3426358757 learned_addr learned my addr 172.21.15.64:0/3426358757 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15f7fe640 1 -- 172.21.15.64:0/3426358757 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 msgr2=0x555f2c001a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2c001a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15f7fe640 1 -- 172.21.15.64:0/3426358757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bff9310 msgr2=0x555f2bff3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bff9310 0x555f2bff3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15f7fe640 1 -- 172.21.15.64:0/3426358757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f2bff3c50 con 0x555f2bfeea50 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff164fd7640 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bff9310 0x555f2bff3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff15f7fe640 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bfeea50 0x555f2bff2f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1540cd540 tx=0x7ff15409b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2a8e5fa3284c06d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff13f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff13f7fe640 1 -- 172.21.15.64:0/3426358757 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1540a44f0 con 0x555f2bfeea50 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1540a4bd0 con 0x555f2bfeea50 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.923+0000 7ff13f7fe640 1 -- 172.21.15.64:0/3426358757 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff15409b160 con 0x555f2bfeea50 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff13f7fe640 1 -- 172.21.15.64:0/3426358757 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1540a4980 con 0x555f2bfeea50 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 10 client.25163 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f2c073d20 con 0x555f2bfeea50 2024-08-19T01:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff13f7fe640 1 -- 172.21.15.64:0/3426358757 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1540a4bd0 con 0x555f2bfeea50 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff13f7fe640 1 -- 172.21.15.64:0/3426358757 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ff1540a1250 con 0x555f2bfeea50 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 10 client.25163 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 10 client.25163 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 20 client.25163 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c073fa0 0x555f2c076390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 4 client.25163 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.2, "await_for": 45.0}] 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.2, "await_for": 45.0}) -- 0x555f2bf5b4f0 con 0x555f2c073fa0 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.924+0000 7ff15ffff640 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c073fa0 0x555f2c076390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.925+0000 7ff15ffff640 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c073fa0 0x555f2c076390 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3335725726f649f9 server_cookie=aae0f79583bbce2f in_seq=0 out_seq=0 2024-08-19T01:39:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.925+0000 7ff13f7fe640 10 client.25163 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.981+0000 7ff13f7fe640 1 -- 172.21.15.64:0/3426358757 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555f2bf5b4f0 con 0x555f2c073fa0 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.981+0000 7ff13f7fe640 10 client.25163 handle_command_reply: tid=0 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.981+0000 7ff168e38740 1 client.25163 shutdown 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bfeea50 msgr2=0x555f2bff2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bfeea50 0x555f2bff2f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1540cd540 tx=0x7ff15409b7a0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 shutdown_connections 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f2bff9310 0x555f2bff3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x555f2c073fa0 0x555f2c076390 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555f2bfeea50 0x555f2bff2f10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff13f7fe640 0 client.25163 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 --2- 172.21.15.64:0/3426358757 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555f2bfee130 0x555f2c001a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 >> 172.21.15.64:0/3426358757 conn(0x555f2bfea6c0 msgr2=0x555f2bfefb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 shutdown_connections 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 1 -- 172.21.15.64:0/3426358757 wait complete. 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff168e38740 20 client.25163 trim_cache size 0 max 16384 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:39:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 18, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "17f16fa1": { 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 18, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 139.2, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:39:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:06.986 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff1480a2cf0 msgr2=0x7ff1480a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff166a3a640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff1480a2cf0 0x7ff1480a51b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x555f2bfeba40 tx=0x7ff158005ef0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1601022a0 msgr2=0x7ff1601995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.982+0000 7ff166a3a640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1601022a0 0x7ff1601995e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff15400e9f0 tx=0x7ff15400edf0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 shutdown_connections 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff1480a2cf0 0x7ff1480a51b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff160102c20 0x7ff1601a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1601022a0 0x7ff1601995e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 --2- 172.21.15.64:0/2055710976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160100b50 0x7ff1601990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 >> 172.21.15.64:0/2055710976 conn(0x7ff1600fc500 msgr2=0x7ff1601065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 shutdown_connections 2024-08-19T01:39:06.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:06.983+0000 7ff166a3a640 1 -- 172.21.15.64:0/2055710976 wait complete. 2024-08-19T01:39:07.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 10.20 MB/sec execute 159 sec latency 407.283 ms 2024-08-19T01:39:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:07 smithi161 ceph-mon[21757]: pgmap v179: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.5 MiB/s wr, 323 op/s 2024-08-19T01:39:07.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:07 smithi064 ceph-mon[18515]: pgmap v179: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.5 MiB/s wr, 323 op/s 2024-08-19T01:39:07.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:07 smithi123 ceph-mon[21829]: pgmap v179: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.5 MiB/s wr, 323 op/s 2024-08-19T01:39:07.839 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '17f16fa1', quiesce took 3.9 seconds. Will release after: 15.299999999999999 2024-08-19T01:39:08.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 10.14 MB/sec execute 160 sec latency 1407.374 ms 2024-08-19T01:39:09.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 10.08 MB/sec execute 161 sec latency 2407.461 ms 2024-08-19T01:39:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:09 smithi064 ceph-mon[18515]: pgmap v180: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-08-19T01:39:09.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:09 smithi123 ceph-mon[21829]: pgmap v180: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-08-19T01:39:09.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:09 smithi161 ceph-mon[21757]: pgmap v180: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-08-19T01:39:10.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 10.02 MB/sec execute 162 sec latency 3407.564 ms 2024-08-19T01:39:11.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.95 MB/sec execute 163 sec latency 4407.618 ms 2024-08-19T01:39:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:11 smithi064 ceph-mon[18515]: pgmap v181: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.8 MiB/s wr, 290 op/s 2024-08-19T01:39:11.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:11 smithi123 ceph-mon[21829]: pgmap v181: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.8 MiB/s wr, 290 op/s 2024-08-19T01:39:11.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:11 smithi161 ceph-mon[21757]: pgmap v181: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.8 MiB/s wr, 290 op/s 2024-08-19T01:39:12.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.89 MB/sec execute 164 sec latency 5407.700 ms 2024-08-19T01:39:13.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.83 MB/sec execute 165 sec latency 6407.816 ms 2024-08-19T01:39:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:13 smithi064 ceph-mon[18515]: pgmap v182: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 186 op/s 2024-08-19T01:39:13.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:13 smithi123 ceph-mon[21829]: pgmap v182: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 186 op/s 2024-08-19T01:39:13.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:13 smithi161 ceph-mon[21757]: pgmap v182: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 186 op/s 2024-08-19T01:39:14.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.77 MB/sec execute 166 sec latency 7407.934 ms 2024-08-19T01:39:15.389 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.72 MB/sec execute 167 sec latency 8408.010 ms 2024-08-19T01:39:15.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:15 smithi064 ceph-mon[18515]: pgmap v183: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 186 op/s 2024-08-19T01:39:15.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:15 smithi123 ceph-mon[21829]: pgmap v183: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 186 op/s 2024-08-19T01:39:15.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:15 smithi161 ceph-mon[21757]: pgmap v183: 129 pgs: 129 active+clean; 355 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 186 op/s 2024-08-19T01:39:16.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.66 MB/sec execute 168 sec latency 9408.133 ms 2024-08-19T01:39:17.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.60 MB/sec execute 169 sec latency 10408.252 ms 2024-08-19T01:39:17.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:17 smithi064 ceph-mon[18515]: pgmap v184: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-08-19T01:39:17.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:17 smithi123 ceph-mon[21829]: pgmap v184: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-08-19T01:39:17.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:17 smithi161 ceph-mon[21757]: pgmap v184: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-08-19T01:39:18.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.54 MB/sec execute 170 sec latency 11408.341 ms 2024-08-19T01:39:19.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.49 MB/sec execute 171 sec latency 12408.462 ms 2024-08-19T01:39:19.467 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:19 smithi123 ceph-mon[21829]: pgmap v185: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 91 op/s 2024-08-19T01:39:19.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:19 smithi064 ceph-mon[18515]: pgmap v185: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 91 op/s 2024-08-19T01:39:19.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:19 smithi161 ceph-mon[21757]: pgmap v185: 129 pgs: 129 active+clean; 348 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 91 op/s 2024-08-19T01:39:20.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.43 MB/sec execute 172 sec latency 13408.545 ms 2024-08-19T01:39:21.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.38 MB/sec execute 173 sec latency 14408.578 ms 2024-08-19T01:39:21.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:21 smithi064 ceph-mon[18515]: pgmap v186: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-08-19T01:39:21.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:21 smithi123 ceph-mon[21829]: pgmap v186: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-08-19T01:39:21.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:21 smithi161 ceph-mon[21757]: pgmap v186: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-08-19T01:39:22.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.33 MB/sec execute 174 sec latency 15408.662 ms 2024-08-19T01:39:22.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:39:22.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:39:22.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:39:22.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:22 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:39:22.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:39:22.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:39:22.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:39:22.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:22 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:39:22.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:39:22.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:39:22.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:39:22.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:22 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:39:23.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:39:23.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.27 MB/sec execute 175 sec latency 16408.703 ms 2024-08-19T01:39:23.407 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:23.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:23 smithi064 ceph-mon[18515]: pgmap v187: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-08-19T01:39:23.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 -- 172.21.15.64:0/2021553380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a41078b0 msgr2=0x7f11a4107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:23.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 --2- 172.21.15.64:0/2021553380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a41078b0 0x7f11a4107c90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1194009cd0 tx=0x7f119402a8e0 comp rx=0 tx=0).stop 2024-08-19T01:39:23.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 -- 172.21.15.64:0/2021553380 shutdown_connections 2024-08-19T01:39:23.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 --2- 172.21.15.64:0/2021553380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a4102250 0x7f11a410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:23.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 --2- 172.21.15.64:0/2021553380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41018b0 0x7f11a4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 --2- 172.21.15.64:0/2021553380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a41078b0 0x7f11a4107c90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 -- 172.21.15.64:0/2021553380 >> 172.21.15.64:0/2021553380 conn(0x7f11a40fd5f0 msgr2=0x7f11a40ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 -- 172.21.15.64:0/2021553380 shutdown_connections 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.870+0000 7f11a9472640 1 -- 172.21.15.64:0/2021553380 wait complete. 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 Processor -- start 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 -- start start 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41018b0 0x7f11a41981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a4102250 0x7f11a4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a41078b0 0x7f11a419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f11a4111cb0 con 0x7f11a4102250 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11a4111b30 con 0x7f11a41018b0 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.871+0000 7f11a9472640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11a4111e30 con 0x7f11a41078b0 2024-08-19T01:39:23.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a4102250 0x7f11a4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a41078b0 0x7f11a419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a4102250 0x7f11a4198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59362/0 (socket says 172.21.15.64:59362) 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 -- 172.21.15.64:0/3163898869 learned_addr learned my addr 172.21.15.64:0/3163898869 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a2ffd640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41018b0 0x7f11a41981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 -- 172.21.15.64:0/3163898869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a41078b0 msgr2=0x7f11a419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a41078b0 0x7f11a419caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 -- 172.21.15.64:0/3163898869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41018b0 msgr2=0x7f11a41981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41018b0 0x7f11a41981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 -- 172.21.15.64:0/3163898869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a419d180 con 0x7f11a4102250 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a37fe640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a41078b0 0x7f11a419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a27fc640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a4102250 0x7f11a4198710 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f118c00d950 tx=0x7f118c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f119bfff640 1 -- 172.21.15.64:0/3163898869 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c017030 con 0x7f11a4102250 2024-08-19T01:39:23.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a419d470 con 0x7f11a4102250 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f119bfff640 1 -- 172.21.15.64:0/3163898869 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f118c004020 con 0x7f11a4102250 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f119bfff640 1 -- 172.21.15.64:0/3163898869 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c0047a0 con 0x7f11a4102250 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.872+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a41a4d50 con 0x7f11a4102250 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.873+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a4101d10 con 0x7f11a4102250 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.874+0000 7f119bfff640 1 -- 172.21.15.64:0/3163898869 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f118c0041d0 con 0x7f11a4102250 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.875+0000 7f119bfff640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f11900a2cf0 0x7f11900a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.876+0000 7f11a2ffd640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f11900a2cf0 0x7f11900a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.876+0000 7f119bfff640 1 -- 172.21.15.64:0/3163898869 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f118c0cf830 con 0x7f11a4102250 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.876+0000 7f11a2ffd640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f11900a2cf0 0x7f11900a51b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1194006010 tx=0x7f1194009360 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:23.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:23.879+0000 7f119bfff640 1 -- 172.21.15.64:0/3163898869 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f118c09c040 con 0x7f11a4102250 2024-08-19T01:39:24.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.088+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11a4199530 con 0x7f11a4102250 2024-08-19T01:39:24.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.088+0000 7f119bfff640 1 -- 172.21.15.64:0/3163898869 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f118c09b940 con 0x7f11a4102250 2024-08-19T01:39:24.090 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:39:24.090 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:39:24.091 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.090+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f11900a2cf0 msgr2=0x7f11900a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.090+0000 7f11a9472640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f11900a2cf0 0x7f11900a51b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1194006010 tx=0x7f1194009360 comp rx=0 tx=0).stop 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.090+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a4102250 msgr2=0x7f11a4198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.090+0000 7f11a9472640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a4102250 0x7f11a4198710 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f118c00d950 tx=0x7f118c00de10 comp rx=0 tx=0).stop 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 shutdown_connections 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f11900a2cf0 0x7f11900a51b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a41078b0 0x7f11a419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11a4102250 0x7f11a4198710 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 --2- 172.21.15.64:0/3163898869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41018b0 0x7f11a41981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 >> 172.21.15.64:0/3163898869 conn(0x7f11a40fd5f0 msgr2=0x7f11a40fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:24.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 shutdown_connections 2024-08-19T01:39:24.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:24.091+0000 7f11a9472640 1 -- 172.21.15.64:0/3163898869 wait complete. 2024-08-19T01:39:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:23 smithi123 ceph-mon[21829]: pgmap v187: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-08-19T01:39:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:23 smithi161 ceph-mon[21757]: pgmap v187: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-08-19T01:39:24.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.22 MB/sec execute 176 sec latency 17408.747 ms 2024-08-19T01:39:24.766 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 17f16fa1 --release --await' 2024-08-19T01:39:24.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 17f16fa1 --release --await 2024-08-19T01:39:24.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:24 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3163898869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:25.025 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:25.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:24 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3163898869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:25.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:24 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3163898869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:25.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 551800 9.17 MB/sec execute 177 sec latency 18408.782 ms 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.519+0000 7f45ab924640 1 -- 172.21.15.64:0/1485828951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4102c20 msgr2=0x7f45a410d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.519+0000 7f45ab924640 1 --2- 172.21.15.64:0/1485828951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4102c20 0x7f45a410d770 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f459c009ca0 tx=0x7f459c02a950 comp rx=0 tx=0).stop 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.519+0000 7f45ab924640 1 -- 172.21.15.64:0/1485828951 shutdown_connections 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 --2- 172.21.15.64:0/1485828951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4102c20 0x7f45a410d770 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 --2- 172.21.15.64:0/1485828951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45a41022a0 0x7f45a41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 --2- 172.21.15.64:0/1485828951 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45a4100b50 0x7f45a4100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 -- 172.21.15.64:0/1485828951 >> 172.21.15.64:0/1485828951 conn(0x7f45a40fc500 msgr2=0x7f45a40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 -- 172.21.15.64:0/1485828951 shutdown_connections 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 -- 172.21.15.64:0/1485828951 wait complete. 2024-08-19T01:39:25.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 Processor -- start 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 -- start start 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4100b50 0x7f45a4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45a41022a0 0x7f45a4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45a4102c20 0x7f45a419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45a4110cb0 con 0x7f45a4100b50 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f45a4110b30 con 0x7f45a4102c20 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.520+0000 7f45ab924640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f45a4110e30 con 0x7f45a41022a0 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4100b50 0x7f45a4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4100b50 0x7f45a4198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59376/0 (socket says 172.21.15.64:59376) 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 -- 172.21.15.64:0/1636212220 learned_addr learned my addr 172.21.15.64:0/1636212220 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45a41022a0 0x7f45a4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45a4102c20 0x7f45a419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 -- 172.21.15.64:0/1636212220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45a41022a0 msgr2=0x7f45a4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45a41022a0 0x7f45a4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 -- 172.21.15.64:0/1636212220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45a4102c20 msgr2=0x7f45a419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45a4102c20 0x7f45a419cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 -- 172.21.15.64:0/1636212220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a419d1e0 con 0x7f45a4100b50 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45a96c0640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4100b50 0x7f45a4198230 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f45a000d770 tx=0x7f45a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f459a7fc640 1 -- 172.21.15.64:0/1636212220 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0017070 con 0x7f45a4100b50 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f459a7fc640 1 -- 172.21.15.64:0/1636212220 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a0005500 con 0x7f45a4100b50 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a419d4d0 con 0x7f45a4100b50 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f459a7fc640 1 -- 172.21.15.64:0/1636212220 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0004e00 con 0x7f45a4100b50 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.521+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a41a4db0 con 0x7f45a4100b50 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f459a7fc640 1 -- 172.21.15.64:0/1636212220 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f45a0004fa0 con 0x7f45a4100b50 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 Processor -- start 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 -- start start 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd404870 0x561ddd404c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f459a7fc640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f45900a2d70 0x7f45900a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd413a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd416370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ddd36ac60 con 0x561ddd405190 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561ddd302b90 con 0x561ddd413f80 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.524+0000 7f45add22740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561ddd366c20 con 0x561ddd404870 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f45900a2d70 0x7f45900a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd413a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f459a7fc640 1 -- 172.21.15.64:0/1636212220 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f45a00cf120 con 0x7f45a4100b50 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd413a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59388/0 (socket says 172.21.15.64:59388) 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 -- 172.21.15.64:0/3838349698 learned_addr learned my addr 172.21.15.64:0/3838349698 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd416370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a96c0640 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd404870 0x561ddd404c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 -- 172.21.15.64:0/3838349698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd404870 msgr2=0x561ddd404c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd404870 0x561ddd404c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 -- 172.21.15.64:0/3838349698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 msgr2=0x561ddd416370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd416370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 -- 172.21.15.64:0/3838349698 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ddd07a580 con 0x561ddd405190 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd413a40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f459400bc30 tx=0x7f45940107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcb524cdbe4ec2d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f45900a2d70 0x7f45900a5230 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f459c005fe0 tx=0x7f459c02ffb0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f458f7fe640 1 -- 172.21.15.64:0/3838349698 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459400be60 con 0x561ddd405190 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f458f7fe640 1 -- 172.21.15.64:0/3838349698 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f459401c960 con 0x561ddd405190 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.525+0000 7f458f7fe640 1 -- 172.21.15.64:0/3838349698 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594011a80 con 0x561ddd405190 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 -- 172.21.15.64:0/3838349698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 msgr2=0x561ddd413a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd413a40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f459400bc30 tx=0x7f45940107a0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 -- 172.21.15.64:0/3838349698 shutdown_connections 2024-08-19T01:39:25.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd416370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd413a40 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 --2- 172.21.15.64:0/3838349698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd404870 0x561ddd404c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 -- 172.21.15.64:0/3838349698 >> 172.21.15.64:0/3838349698 conn(0x561ddd400f20 msgr2=0x561ddd401300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 -- 172.21.15.64:0/3838349698 shutdown_connections 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 -- 172.21.15.64:0/3838349698 wait complete. 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.526+0000 7f45add22740 1 -- start start 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 1 Processor -- start 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd404870 0x561ddd410f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd411460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd4119a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ddd416b50 con 0x561ddd405190 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561ddd3087c0 con 0x561ddd404870 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561ddd416cd0 con 0x561ddd413f80 2024-08-19T01:39:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45add22740 10 client.0 _ll_register_callbacks cb 0x561ddd311f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd411460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd411460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59402/0 (socket says 172.21.15.64:59402) 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 -- 172.21.15.64:0/205993872 learned_addr learned my addr 172.21.15.64:0/205993872 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a96c0640 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd404870 0x561ddd410f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd4119a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 -- 172.21.15.64:0/205993872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 msgr2=0x561ddd4119a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd4119a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 -- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd404870 msgr2=0x561ddd410f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd404870 0x561ddd410f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.527+0000 7f45a8ebf640 1 -- 172.21.15.64:0/205993872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ddd144960 con 0x561ddd405190 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd411460 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f459400b5f0 tx=0x7f459400be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a29c2722ead721 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f458cff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:39:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f458cff9640 1 -- 172.21.15.64:0/205993872 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459402b060 con 0x561ddd405190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f458cff9640 1 -- 172.21.15.64:0/205993872 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f459401c690 con 0x561ddd405190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f459401c960 con 0x561ddd405190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f458cff9640 1 -- 172.21.15.64:0/205993872 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4594032050 con 0x561ddd405190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45add22740 10 client.15705 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561ddd422cd0 con 0x561ddd405190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f458cff9640 1 -- 172.21.15.64:0/205993872 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f459401c960 con 0x561ddd405190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f458cff9640 1 -- 172.21.15.64:0/205993872 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f4594011810 con 0x561ddd405190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45add22740 10 client.15705 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45add22740 10 client.15705 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45add22740 20 client.15705 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.528+0000 7f45add22740 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd424190 0x561ddd426580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.529+0000 7f45add22740 4 client.15705 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.529+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ddd3781d0 con 0x561ddd424190 2024-08-19T01:39:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.529+0000 7f45a96c0640 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd424190 0x561ddd426580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.529+0000 7f458cff9640 10 client.15705 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.529+0000 7f45a96c0640 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd424190 0x561ddd426580 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54b1f5e20c4516cd server_cookie=6e318597b1aedd91 in_seq=0 out_seq=0 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f458cff9640 1 -- 172.21.15.64:0/205993872 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ddd3781d0 con 0x561ddd424190 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f458cff9640 10 client.15705 handle_command_reply: tid=0 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 client.15705 shutdown 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 msgr2=0x561ddd411460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd411460 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f459400b5f0 tx=0x7f459400be90 comp rx=0 tx=0).stop 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 shutdown_connections 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd4119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd405190 0x561ddd411460 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd424190 0x561ddd426580 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 --2- 172.21.15.64:0/205993872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd404870 0x561ddd410f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 >> 172.21.15.64:0/205993872 conn(0x561ddd400f20 msgr2=0x561ddd406c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:25.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f458cff9640 0 client.15705 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 shutdown_connections 2024-08-19T01:39:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.530+0000 7f45add22740 1 -- 172.21.15.64:0/205993872 wait complete. 2024-08-19T01:39:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.531+0000 7f45add22740 20 client.15705 trim_cache size 0 max 16384 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 Processor -- start 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 -- start start 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd40fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd405190 0x561ddd42bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd42c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41af420 con 0x561ddd404870 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41accf0 con 0x561ddd413f80 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45add22740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41aab60 con 0x561ddd405190 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd40fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd40fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59418/0 (socket says 172.21.15.64:59418) 2024-08-19T01:39:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 -- 172.21.15.64:0/2635766536 learned_addr learned my addr 172.21.15.64:0/2635766536 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd405190 0x561ddd42bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd42c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 -- 172.21.15.64:0/2635766536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd405190 msgr2=0x561ddd42bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd405190 0x561ddd42bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 -- 172.21.15.64:0/2635766536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 msgr2=0x561ddd42c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd42c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.548+0000 7f45a96c0640 1 -- 172.21.15.64:0/2635766536 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ddd3ff470 con 0x561ddd404870 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd42c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd405190 0x561ddd42bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f45a96c0640 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd40fe60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f45a00cc520 tx=0x7f45a009e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4fb9033ab906ad server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f458effd640 1 -- 172.21.15.64:0/2635766536 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00b4070 con 0x561ddd404870 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f458effd640 1 -- 172.21.15.64:0/2635766536 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a009f640 con 0x561ddd404870 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f458effd640 1 -- 172.21.15.64:0/2635766536 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0020040 con 0x561ddd404870 2024-08-19T01:39:25.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f45add22740 1 -- 172.21.15.64:0/2635766536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 msgr2=0x561ddd40fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.549+0000 7f45add22740 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd40fe60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f45a00cc520 tx=0x7f45a009e040 comp rx=0 tx=0).stop 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- 172.21.15.64:0/2635766536 shutdown_connections 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd413f80 0x561ddd42c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd405190 0x561ddd42bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 --2- 172.21.15.64:0/2635766536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd40fe60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- 172.21.15.64:0/2635766536 >> 172.21.15.64:0/2635766536 conn(0x561ddd400f20 msgr2=0x561ddd412fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- 172.21.15.64:0/2635766536 shutdown_connections 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- 172.21.15.64:0/2635766536 wait complete. 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- start start 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 Processor -- start 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd41c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd405190 0x561ddd4176d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd417c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41ab1b0 con 0x561ddd404870 2024-08-19T01:39:25.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41ac620 con 0x561ddd405190 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41ab880 con 0x561ddd413f80 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.550+0000 7f45add22740 10 client.0 _ll_register_callbacks cb 0x7f4594011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd41c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd41c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59424/0 (socket says 172.21.15.64:59424) 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 -- 172.21.15.64:0/759152870 learned_addr learned my addr 172.21.15.64:0/759152870 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a9ec1640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd417c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd405190 0x561ddd4176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 -- 172.21.15.64:0/759152870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 msgr2=0x561ddd417c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd417c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 -- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd405190 msgr2=0x561ddd4176d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd405190 0x561ddd4176d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 -- 172.21.15.64:0/759152870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ddd418150 con 0x561ddd404870 2024-08-19T01:39:25.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd405190 0x561ddd4176d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45a96c0640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd41c3a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f45a009b2d0 tx=0x7f45a000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3260bcaa10abc93 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f458cff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f458cff9640 1 -- 172.21.15.64:0/759152870 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00b9060 con 0x561ddd404870 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f458cff9640 1 -- 172.21.15.64:0/759152870 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f45a0099640 con 0x561ddd404870 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a009f640 con 0x561ddd404870 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f458cff9640 1 -- 172.21.15.64:0/759152870 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f45a0020070 con 0x561ddd404870 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45add22740 10 client.15711 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.551+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561ddd48e790 con 0x561ddd404870 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f458cff9640 1 -- 172.21.15.64:0/759152870 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a009f640 con 0x561ddd404870 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f458cff9640 1 -- 172.21.15.64:0/759152870 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f45a00cc740 con 0x561ddd404870 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45add22740 10 client.15711 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45add22740 10 client.15711 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45add22740 20 client.15711 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45add22740 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd48ee50 0x561ddd491270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:25.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45add22740 4 client.15711 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "17f16fa1", "release": true, "await": true}] 2024-08-19T01:39:25.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "17f16fa1", "release": true, "await": true}) -- 0x561ddd3726e0 con 0x561ddd48ee50 2024-08-19T01:39:25.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd48ee50 0x561ddd491270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:25.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f45a8ebf640 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd48ee50 0x561ddd491270 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ecca4ea7f8e094e server_cookie=4216fb565d42f33f in_seq=0 out_seq=0 2024-08-19T01:39:25.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.552+0000 7f458cff9640 10 client.15711 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f458cff9640 1 -- 172.21.15.64:0/759152870 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561ddd3726e0 con 0x561ddd48ee50 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f458cff9640 10 client.15711 handle_command_reply: tid=0 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 client.15711 shutdown 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 msgr2=0x561ddd41c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd41c3a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f45a009b2d0 tx=0x7f45a000b480 comp rx=0 tx=0).stop 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 shutdown_connections 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561ddd413f80 0x561ddd417c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561ddd48ee50 0x561ddd491270 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561ddd405190 0x561ddd4176d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 --2- 172.21.15.64:0/759152870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ddd404870 0x561ddd41c3a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 >> 172.21.15.64:0/759152870 conn(0x561ddd400f20 msgr2=0x561ddd3ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.559+0000 7f458cff9640 0 client.15711 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:25.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 shutdown_connections 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45add22740 1 -- 172.21.15.64:0/759152870 wait complete. 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45add22740 20 client.15711 trim_cache size 0 max 16384 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 20, 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:39:25.562 INFO:teuthology.orchestra.run.smithi064.stdout: "17f16fa1": { 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 20, 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 139.2, 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:25.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:25.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:25.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:25.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:25.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:25.564 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:39:25.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f45900a2d70 msgr2=0x7f45900a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f45900a2d70 0x7f45900a5230 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f459c005fe0 tx=0x7f459c02ffb0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4100b50 msgr2=0x7f45a4198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4100b50 0x7f45a4198230 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f45a000d770 tx=0x7f45a000dc30 comp rx=0 tx=0).stop 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 shutdown_connections 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f45900a2d70 0x7f45900a5230 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45a4102c20 0x7f45a419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45a41022a0 0x7f45a4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 --2- 172.21.15.64:0/1636212220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a4100b50 0x7f45a4198230 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.560+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 >> 172.21.15.64:0/1636212220 conn(0x7f45a40fc500 msgr2=0x7f45a4069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.561+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 shutdown_connections 2024-08-19T01:39:25.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:25.561+0000 7f45ab924640 1 -- 172.21.15.64:0/1636212220 wait complete. 2024-08-19T01:39:25.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:25 smithi064 ceph-mon[18515]: pgmap v188: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-08-19T01:39:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:25 smithi123 ceph-mon[21829]: pgmap v188: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-08-19T01:39:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:25 smithi161 ceph-mon[21757]: pgmap v188: 129 pgs: 129 active+clean; 347 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 60 op/s 2024-08-19T01:39:26.349 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '17f16fa1', total seconds elapsed: 22.4 2024-08-19T01:39:26.349 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.2 2024-08-19T01:39:26.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:39:26.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 554159 9.20 MB/sec execute 178 sec latency 18579.402 ms 2024-08-19T01:39:26.710 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:27.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1700049978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c1034b0 msgr2=0x7f739c1078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:27.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1700049978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c1034b0 0x7f739c1078f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7388009930 tx=0x7f738802a420 comp rx=0 tx=0).stop 2024-08-19T01:39:27.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1700049978 shutdown_connections 2024-08-19T01:39:27.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1700049978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f739c108030 0x7f739c10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1700049978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c1034b0 0x7f739c1078f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1700049978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f739c102b00 0x7f739c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1700049978 >> 172.21.15.64:0/1700049978 conn(0x7f739c0fc500 msgr2=0x7f739c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1700049978 shutdown_connections 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.152+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1700049978 wait complete. 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.153+0000 7f73a2a6b640 1 Processor -- start 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.153+0000 7f73a2a6b640 1 -- start start 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a2a6b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c102b00 0x7f739c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a2a6b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f739c1034b0 0x7f739c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a2a6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f739c108030 0x7f739c19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a2a6b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f739c111e50 con 0x7f739c102b00 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a2a6b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f739c111cd0 con 0x7f739c1034b0 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a2a6b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f739c111fd0 con 0x7f739c108030 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c102b00 0x7f739c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c102b00 0x7f739c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:59446/0 (socket says 172.21.15.64:59446) 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 -- 172.21.15.64:0/1240908136 learned_addr learned my addr 172.21.15.64:0/1240908136 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f739bfff640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f739c1034b0 0x7f739c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a1008640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f739c108030 0x7f739c19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 -- 172.21.15.64:0/1240908136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f739c108030 msgr2=0x7f739c19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f739c108030 0x7f739c19cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 -- 172.21.15.64:0/1240908136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f739c1034b0 msgr2=0x7f739c198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f739c1034b0 0x7f739c198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 -- 172.21.15.64:0/1240908136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c19d2b0 con 0x7f739c102b00 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f73a0807640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c102b00 0x7f739c198300 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f739000ea60 tx=0x7f739000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.154+0000 7f7399ffb640 1 -- 172.21.15.64:0/1240908136 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739000cde0 con 0x7f739c102b00 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.155+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c19d5a0 con 0x7f739c102b00 2024-08-19T01:39:27.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.155+0000 7f7399ffb640 1 -- 172.21.15.64:0/1240908136 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73900035f0 con 0x7f739c102b00 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.155+0000 7f7399ffb640 1 -- 172.21.15.64:0/1240908136 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390010670 con 0x7f739c102b00 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.155+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c1a4e80 con 0x7f739c102b00 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.156+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739c102ee0 con 0x7f739c102b00 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.157+0000 7f7399ffb640 1 -- 172.21.15.64:0/1240908136 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f73900037a0 con 0x7f739c102b00 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.157+0000 7f7399ffb640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f738c0a2cf0 0x7f738c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.157+0000 7f739bfff640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f738c0a2cf0 0x7f738c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.158+0000 7f7399ffb640 1 -- 172.21.15.64:0/1240908136 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f73900cfab0 con 0x7f739c102b00 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.158+0000 7f739bfff640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f738c0a2cf0 0x7f738c0a51b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f738802a940 tx=0x7f7388005a40 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:27.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.162+0000 7f7399ffb640 1 -- 172.21.15.64:0/1240908136 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f739009c320 con 0x7f739c102b00 2024-08-19T01:39:27.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.383+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f739c10a010 con 0x7f739c102b00 2024-08-19T01:39:27.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.384+0000 7f7399ffb640 1 -- 172.21.15.64:0/1240908136 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f739009bc20 con 0x7f739c102b00 2024-08-19T01:39:27.385 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:39:27.385 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:39:27.385 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:39:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f738c0a2cf0 msgr2=0x7f738c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f738c0a2cf0 0x7f738c0a51b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f738802a940 tx=0x7f7388005a40 comp rx=0 tx=0).stop 2024-08-19T01:39:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c102b00 msgr2=0x7f739c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c102b00 0x7f739c198300 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f739000ea60 tx=0x7f739000ef20 comp rx=0 tx=0).stop 2024-08-19T01:39:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 shutdown_connections 2024-08-19T01:39:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f738c0a2cf0 0x7f738c0a51b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f739c108030 0x7f739c19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f739c1034b0 0x7f739c198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 --2- 172.21.15.64:0/1240908136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f739c102b00 0x7f739c198300 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:27.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 >> 172.21.15.64:0/1240908136 conn(0x7f739c0fc500 msgr2=0x7f739c105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:27.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 shutdown_connections 2024-08-19T01:39:27.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:27.386+0000 7f73a2a6b640 1 -- 172.21.15.64:0/1240908136 wait complete. 2024-08-19T01:39:27.390 INFO:tasks.workunit.client.0.smithi064.stdout: 1 555769 9.19 MB/sec execute 179 sec latency 36.094 ms 2024-08-19T01:39:28.035 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:27 smithi064 ceph-mon[18515]: pgmap v189: 129 pgs: 129 active+clean; 346 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-08-19T01:39:28.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:27 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1240908136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:28.063 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 139.2 --await-for 45.0' 2024-08-19T01:39:28.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 139.2 --await-for 45.0 2024-08-19T01:39:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:27 smithi123 ceph-mon[21829]: pgmap v189: 129 pgs: 129 active+clean; 346 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-08-19T01:39:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:27 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1240908136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:27 smithi161 ceph-mon[21757]: pgmap v189: 129 pgs: 129 active+clean; 346 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-08-19T01:39:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:27 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1240908136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:28.326 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:28.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 558372 9.22 MB/sec execute 180 sec latency 31.025 ms 2024-08-19T01:39:28.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 -- 172.21.15.64:0/2825129957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d4102260 msgr2=0x7fb3d410e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2825129957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d4102260 0x7fb3d410e790 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009cd0 tx=0x7fb3cc02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 -- 172.21.15.64:0/2825129957 shutdown_connections 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2825129957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d4102260 0x7fb3d410e790 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2825129957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3d41018c0 0x7fb3d4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2825129957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb3d41078c0 0x7fb3d4107ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 -- 172.21.15.64:0/2825129957 >> 172.21.15.64:0/2825129957 conn(0x7fb3d40fd5e0 msgr2=0x7fb3d40ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 -- 172.21.15.64:0/2825129957 shutdown_connections 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.825+0000 7fb3daf24640 1 -- 172.21.15.64:0/2825129957 wait complete. 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 Processor -- start 2024-08-19T01:39:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 -- start start 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3d41018c0 0x7fb3d4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb3d4102260 0x7fb3d41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d41078c0 0x7fb3d419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d4111cc0 con 0x7fb3d41078c0 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d4111b40 con 0x7fb3d41018c0 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3daf24640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d4111e40 con 0x7fb3d4102260 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3d94c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d41078c0 0x7fb3d419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3d8cc0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3d41018c0 0x7fb3d4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3d94c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d41078c0 0x7fb3d419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34360/0 (socket says 172.21.15.64:34360) 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3d94c1640 1 -- 172.21.15.64:0/2817799658 learned_addr learned my addr 172.21.15.64:0/2817799658 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.826+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb3d4102260 0x7fb3d41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/2817799658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb3d4102260 msgr2=0x7fb3d41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb3d4102260 0x7fb3d41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d41078c0 msgr2=0x7fb3d419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d41078c0 0x7fb3d419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/2817799658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d419d130 con 0x7fb3d41018c0 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d41078c0 0x7fb3d419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3d41018c0 0x7fb3d4198180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800e9f0 tx=0x7fb3c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d1ffb640 1 -- 172.21.15.64:0/2817799658 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c800ccc0 con 0x7fb3d41018c0 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d419d420 con 0x7fb3d41018c0 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d1ffb640 1 -- 172.21.15.64:0/2817799658 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c8005020 con 0x7fb3d41018c0 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3d1ffb640 1 -- 172.21.15.64:0/2817799658 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80049e0 con 0x7fb3d41018c0 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.827+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d41a4d00 con 0x7fb3d41018c0 2024-08-19T01:39:28.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.829+0000 7fb3d1ffb640 1 -- 172.21.15.64:0/2817799658 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb3c8005560 con 0x7fb3d41018c0 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 Processor -- start 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 -- start start 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d1ffb640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb3c40a2cf0 0x7fb3c40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb3c40a2cf0 0x7fb3c40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98b130 0x55d8ec98b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec995dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d1ffb640 1 -- 172.21.15.64:0/2817799658 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb3c80cf750 con 0x7fb3d41018c0 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec998700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d8ec8f0a70 con 0x55d8ec98ba50 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55d8ec888780 con 0x55d8ec98b130 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3dd322740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d8ec8eca30 con 0x55d8ec996310 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d8cc0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98b130 0x55d8ec98b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d8cc0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98b130 0x55d8ec98b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:54008/0 (socket says 172.21.15.64:54008) 2024-08-19T01:39:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d94c1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec998700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1213336212 learned_addr learned my addr 172.21.15.64:0/1213336212 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.830+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec995dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1213336212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 msgr2=0x55d8ec998700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec998700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1213336212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 msgr2=0x55d8ec995dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec995dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1213336212 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8ec662b90 con 0x55d8ec98b130 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb3c40a2cf0 0x7fb3c40a51b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc031860 tx=0x7fb3cc031b50 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:28.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98b130 0x55d8ec98b510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80a3fa0 tx=0x7fb3c8005290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffad97213c87ac95 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3beffd640 1 -- 172.21.15.64:0/1213336212 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c801c040 con 0x55d8ec98b130 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3beffd640 1 -- 172.21.15.64:0/1213336212 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c809fc10 con 0x55d8ec98b130 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.831+0000 7fb3beffd640 1 -- 172.21.15.64:0/1213336212 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8086070 con 0x55d8ec98b130 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 -- 172.21.15.64:0/1213336212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98b130 msgr2=0x55d8ec98b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98b130 0x55d8ec98b510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80a3fa0 tx=0x7fb3c8005290 comp rx=0 tx=0).stop 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 -- 172.21.15.64:0/1213336212 shutdown_connections 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec998700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec995dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1213336212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98b130 0x55d8ec98b510 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 -- 172.21.15.64:0/1213336212 >> 172.21.15.64:0/1213336212 conn(0x55d8ec9876c0 msgr2=0x55d8ec987aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 -- 172.21.15.64:0/1213336212 shutdown_connections 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 -- 172.21.15.64:0/1213336212 wait complete. 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 -- start start 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.832+0000 7fb3dd322740 1 Processor -- start 2024-08-19T01:39:28.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98b130 0x55d8ec993200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98ba50 0x55d8ec993740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec993c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3dd322740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d8ec998ee0 con 0x55d8ec98b130 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3dd322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55d8ec8ab990 con 0x55d8ec98ba50 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3dd322740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d8ec999060 con 0x55d8ec996310 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3dd322740 10 client.0 _ll_register_callbacks cb 0x55d8ec600090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98b130 0x55d8ec993200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98b130 0x55d8ec993200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34372/0 (socket says 172.21.15.64:34372) 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1906859174 learned_addr learned my addr 172.21.15.64:0/1906859174 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98ba50 0x55d8ec993740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec993c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1906859174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 msgr2=0x55d8ec993c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec993c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98ba50 msgr2=0x55d8ec993740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98ba50 0x55d8ec993740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 -- 172.21.15.64:0/1906859174 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8ec8aa6c0 con 0x55d8ec98b130 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98ba50 0x55d8ec993740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98b130 0x55d8ec993200 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8098230 tx=0x7fb3c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60179c0fdbe5062 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.833+0000 7fb3b3fff640 1 -- 172.21.15.64:0/1906859174 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80b8020 con 0x55d8ec98b130 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3c809fc10 con 0x55d8ec98b130 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3b3fff640 1 -- 172.21.15.64:0/1906859174 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb3c80860c0 con 0x55d8ec98b130 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3b3fff640 1 -- 172.21.15.64:0/1906859174 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb3c801c040 con 0x55d8ec98b130 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 10 client.15729 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d8ec9a4b90 con 0x55d8ec98b130 2024-08-19T01:39:28.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3b3fff640 1 -- 172.21.15.64:0/1906859174 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c809fc10 con 0x55d8ec98b130 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3b3fff640 1 -- 172.21.15.64:0/1906859174 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb3c80a4040 con 0x55d8ec98b130 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 10 client.15729 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 10 client.15729 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 20 client.15729 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8ec9a5ee0 0x55d8ec9a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 4 client.15729 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8ec8fe1a0 con 0x55d8ec9a5ee0 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.834+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8ec9a5ee0 0x55d8ec9a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.835+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8ec9a5ee0 0x55d8ec9a82d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58a180b1c38baaf7 server_cookie=543240320d61816a in_seq=0 out_seq=0 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.835+0000 7fb3b3fff640 10 client.15729 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.835+0000 7fb3b3fff640 1 -- 172.21.15.64:0/1906859174 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d8ec8fe1a0 con 0x55d8ec9a5ee0 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.835+0000 7fb3b3fff640 10 client.15729 handle_command_reply: tid=0 2024-08-19T01:39:28.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.835+0000 7fb3dd322740 1 client.15729 shutdown 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98b130 msgr2=0x55d8ec993200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98b130 0x55d8ec993200 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8098230 tx=0x7fb3c80046c0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 shutdown_connections 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec996310 0x55d8ec993c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8ec9a5ee0 0x55d8ec9a82d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec98ba50 0x55d8ec993740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 --2- 172.21.15.64:0/1906859174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98b130 0x55d8ec993200 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3b3fff640 0 client.15729 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 >> 172.21.15.64:0/1906859174 conn(0x55d8ec9876c0 msgr2=0x55d8ec98d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 shutdown_connections 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 1 -- 172.21.15.64:0/1906859174 wait complete. 2024-08-19T01:39:28.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.836+0000 7fb3dd322740 20 client.15729 trim_cache size 0 max 16384 2024-08-19T01:39:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 Processor -- start 2024-08-19T01:39:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 -- start start 2024-08-19T01:39:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec992140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec9ad900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41b1780 con 0x55d8ec98ba50 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41af050 con 0x55d8ec996310 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3dd322740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41acea0 con 0x55d8ec98b130 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34384/0 (socket says 172.21.15.64:34384) 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3d94c1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec9ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3d3fff640 1 -- 172.21.15.64:0/2757388255 learned_addr learned my addr 172.21.15.64:0/2757388255 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.855+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec992140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d94c1640 1 -- 172.21.15.64:0/2757388255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 msgr2=0x55d8ec992140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec992140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d94c1640 1 -- 172.21.15.64:0/2757388255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 msgr2=0x55d8ec98db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d94c1640 1 -- 172.21.15.64:0/2757388255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8ec988770 con 0x55d8ec996310 2024-08-19T01:39:28.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec992140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:28.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:28.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec9ad900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009ca0 tx=0x7fb3cc049000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92071cabd9bcaf0d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:28.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3be7fc640 1 -- 172.21.15.64:0/2757388255 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc044040 con 0x55d8ec996310 2024-08-19T01:39:28.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3be7fc640 1 -- 172.21.15.64:0/2757388255 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc04e030 con 0x55d8ec996310 2024-08-19T01:39:28.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.856+0000 7fb3be7fc640 1 -- 172.21.15.64:0/2757388255 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc04a6b0 con 0x55d8ec996310 2024-08-19T01:39:28.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 -- 172.21.15.64:0/2757388255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 msgr2=0x55d8ec9ad900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec9ad900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009ca0 tx=0x7fb3cc049000 comp rx=0 tx=0).stop 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 -- 172.21.15.64:0/2757388255 shutdown_connections 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec9ad900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 --2- 172.21.15.64:0/2757388255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec992140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 -- 172.21.15.64:0/2757388255 >> 172.21.15.64:0/2757388255 conn(0x55d8ec9876c0 msgr2=0x55d8ec995280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 -- 172.21.15.64:0/2757388255 shutdown_connections 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 -- 172.21.15.64:0/2757388255 wait complete. 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 -- start start 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.857+0000 7fb3dd322740 1 Processor -- start 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec99eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3dd322740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec990540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3dd322740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41ad510 con 0x55d8ec98ba50 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3dd322740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41ae980 con 0x55d8ec996310 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3dd322740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41adbe0 con 0x55d8ec98b130 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3dd322740 10 client.0 _ll_register_callbacks cb 0x7fb3c00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d8cc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec99eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34386/0 (socket says 172.21.15.64:34386) 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 -- 172.21.15.64:0/3273660366 learned_addr learned my addr 172.21.15.64:0/3273660366 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec990540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 -- 172.21.15.64:0/3273660366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 msgr2=0x55d8ec99eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec99eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 -- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 msgr2=0x55d8ec990540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec990540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d3fff640 1 -- 172.21.15.64:0/3273660366 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8ec990d70 con 0x55d8ec98ba50 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d94c1640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec990540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.858+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec99eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3d3fff640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98ffc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x55d8ec992a90 tx=0x7fb3c0012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f6a0e1e1a1f7d0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3b3fff640 1 -- 172.21.15.64:0/3273660366 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0004230 con 0x55d8ec98ba50 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3b3fff640 1 -- 172.21.15.64:0/3273660366 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb3c00043d0 con 0x55d8ec98ba50 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3cc04e030 con 0x55d8ec98ba50 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3b3fff640 1 -- 172.21.15.64:0/3273660366 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb3c0005020 con 0x55d8ec98ba50 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 10 client.15735 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d8eca109b0 con 0x55d8ec98ba50 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3b3fff640 1 -- 172.21.15.64:0/3273660366 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc04e030 con 0x55d8ec98ba50 2024-08-19T01:39:28.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3b3fff640 1 -- 172.21.15.64:0/3273660366 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb3c0010610 con 0x55d8ec98ba50 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 10 client.15735 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 10 client.15735 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 20 client.15735 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8eca111d0 0x55d8eca135f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 4 client.15735 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.2, "await_for": 45.0}] 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.859+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.2, "await_for": 45.0}) -- 0x55d8ec8f84f0 con 0x55d8eca111d0 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.860+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8eca111d0 0x55d8eca135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.860+0000 7fb3b3fff640 10 client.15735 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:28.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.860+0000 7fb3d8cc0640 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8eca111d0 0x55d8eca135f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a216277230c06b0 server_cookie=8685ddc1632e3251 in_seq=0 out_seq=0 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3b3fff640 1 -- 172.21.15.64:0/3273660366 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d8ec8f84f0 con 0x55d8eca111d0 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3b3fff640 10 client.15735 handle_command_reply: tid=0 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 client.15735 shutdown 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 msgr2=0x55d8ec98ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98ffc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x55d8ec992a90 tx=0x7fb3c0012ae0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 shutdown_connections 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55d8eca111d0 0x55d8eca135f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55d8ec996310 0x55d8ec990540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8ec98ba50 0x55d8ec98ffc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3b3fff640 0 client.15735 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:28.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 --2- 172.21.15.64:0/3273660366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d8ec98b130 0x55d8ec99eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.920+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 >> 172.21.15.64:0/3273660366 conn(0x55d8ec9876c0 msgr2=0x55d8ec985ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:28.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 shutdown_connections 2024-08-19T01:39:28.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3dd322740 1 -- 172.21.15.64:0/3273660366 wait complete. 2024-08-19T01:39:28.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3dd322740 20 client.15735 trim_cache size 0 max 16384 2024-08-19T01:39:28.923 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:39:28.924 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:39:28.924 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:39:28.924 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 22, 2024-08-19T01:39:28.924 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:39:28.924 INFO:teuthology.orchestra.run.smithi064.stdout: "dfc10a11": { 2024-08-19T01:39:28.924 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 22, 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 139.2, 2024-08-19T01:39:28.925 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:28.926 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:39:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb3c40a2cf0 msgr2=0x7fb3c40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb3c40a2cf0 0x7fb3c40a51b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc031860 tx=0x7fb3cc031b50 comp rx=0 tx=0).stop 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3d41018c0 msgr2=0x7fb3d4198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3d41018c0 0x7fb3d4198180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800e9f0 tx=0x7fb3c800eeb0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 shutdown_connections 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb3c40a2cf0 0x7fb3c40a51b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3d41078c0 0x7fb3d419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb3d4102260 0x7fb3d41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 --2- 172.21.15.64:0/2817799658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3d41018c0 0x7fb3d4198180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 >> 172.21.15.64:0/2817799658 conn(0x7fb3d40fd5e0 msgr2=0x7fb3d40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 shutdown_connections 2024-08-19T01:39:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:28.921+0000 7fb3daf24640 1 -- 172.21.15.64:0/2817799658 wait complete. 2024-08-19T01:39:29.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 9.19 MB/sec execute 181 sec latency 470.942 ms 2024-08-19T01:39:29.739 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'dfc10a11', quiesce took 3.4 seconds. Will release after: 15.799999999999999 2024-08-19T01:39:29.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:29 smithi064 ceph-mon[18515]: pgmap v190: 129 pgs: 129 active+clean; 346 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.8 MiB/s wr, 67 op/s 2024-08-19T01:39:30.001 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:29 smithi161 ceph-mon[21757]: pgmap v190: 129 pgs: 129 active+clean; 346 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.8 MiB/s wr, 67 op/s 2024-08-19T01:39:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:29 smithi123 ceph-mon[21829]: pgmap v190: 129 pgs: 129 active+clean; 346 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 1.8 MiB/s wr, 67 op/s 2024-08-19T01:39:30.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 9.14 MB/sec execute 182 sec latency 1470.986 ms 2024-08-19T01:39:31.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 9.09 MB/sec execute 183 sec latency 2471.094 ms 2024-08-19T01:39:32.070 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:31 smithi064 ceph-mon[18515]: pgmap v191: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 150 op/s 2024-08-19T01:39:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:31 smithi123 ceph-mon[21829]: pgmap v191: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 150 op/s 2024-08-19T01:39:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:31 smithi161 ceph-mon[21757]: pgmap v191: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 150 op/s 2024-08-19T01:39:32.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 9.04 MB/sec execute 184 sec latency 3471.157 ms 2024-08-19T01:39:33.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.99 MB/sec execute 185 sec latency 4471.233 ms 2024-08-19T01:39:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:33 smithi123 ceph-mon[21829]: pgmap v192: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 116 op/s 2024-08-19T01:39:34.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:33 smithi161 ceph-mon[21757]: pgmap v192: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 116 op/s 2024-08-19T01:39:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:33 smithi064 ceph-mon[18515]: pgmap v192: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 116 op/s 2024-08-19T01:39:34.391 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.95 MB/sec execute 186 sec latency 5471.706 ms 2024-08-19T01:39:35.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.90 MB/sec execute 187 sec latency 6471.787 ms 2024-08-19T01:39:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:35 smithi123 ceph-mon[21829]: pgmap v193: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 116 op/s 2024-08-19T01:39:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:35 smithi161 ceph-mon[21757]: pgmap v193: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 116 op/s 2024-08-19T01:39:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:35 smithi064 ceph-mon[18515]: pgmap v193: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 116 op/s 2024-08-19T01:39:36.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.85 MB/sec execute 188 sec latency 7471.906 ms 2024-08-19T01:39:37.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.80 MB/sec execute 189 sec latency 8472.033 ms 2024-08-19T01:39:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:37 smithi123 ceph-mon[21829]: pgmap v194: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 150 op/s 2024-08-19T01:39:38.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:37 smithi161 ceph-mon[21757]: pgmap v194: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 150 op/s 2024-08-19T01:39:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:37 smithi064 ceph-mon[18515]: pgmap v194: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 150 op/s 2024-08-19T01:39:38.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.76 MB/sec execute 190 sec latency 9472.145 ms 2024-08-19T01:39:39.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.71 MB/sec execute 191 sec latency 10472.258 ms 2024-08-19T01:39:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:39 smithi123 ceph-mon[21829]: pgmap v195: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-08-19T01:39:40.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:39 smithi161 ceph-mon[21757]: pgmap v195: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-08-19T01:39:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:39 smithi064 ceph-mon[18515]: pgmap v195: 129 pgs: 129 active+clean; 351 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-08-19T01:39:40.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.67 MB/sec execute 192 sec latency 11472.364 ms 2024-08-19T01:39:41.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.62 MB/sec execute 193 sec latency 12472.469 ms 2024-08-19T01:39:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:41 smithi123 ceph-mon[21829]: pgmap v196: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 142 op/s 2024-08-19T01:39:42.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:41 smithi161 ceph-mon[21757]: pgmap v196: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 142 op/s 2024-08-19T01:39:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:41 smithi064 ceph-mon[18515]: pgmap v196: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 142 op/s 2024-08-19T01:39:42.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.58 MB/sec execute 194 sec latency 13472.565 ms 2024-08-19T01:39:43.392 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.53 MB/sec execute 195 sec latency 14472.658 ms 2024-08-19T01:39:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:43 smithi123 ceph-mon[21829]: pgmap v197: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.6 MiB/s wr, 59 op/s 2024-08-19T01:39:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:43 smithi161 ceph-mon[21757]: pgmap v197: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.6 MiB/s wr, 59 op/s 2024-08-19T01:39:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:43 smithi064 ceph-mon[18515]: pgmap v197: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.6 MiB/s wr, 59 op/s 2024-08-19T01:39:44.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.49 MB/sec execute 196 sec latency 15472.701 ms 2024-08-19T01:39:45.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.45 MB/sec execute 197 sec latency 16472.786 ms 2024-08-19T01:39:45.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:39:45.815 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:45.938 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:45 smithi064 ceph-mon[18515]: pgmap v198: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.6 MiB/s wr, 59 op/s 2024-08-19T01:39:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:45 smithi123 ceph-mon[21829]: pgmap v198: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.6 MiB/s wr, 59 op/s 2024-08-19T01:39:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:45 smithi161 ceph-mon[21757]: pgmap v198: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.6 MiB/s wr, 59 op/s 2024-08-19T01:39:46.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 -- 172.21.15.64:0/990784641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c100b50 msgr2=0x7f3f7c100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:46.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/990784641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c100b50 0x7f3f7c100f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3f78009900 tx=0x7f3f7802a480 comp rx=0 tx=0).stop 2024-08-19T01:39:46.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 -- 172.21.15.64:0/990784641 shutdown_connections 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/990784641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7c102c20 0x7f3f7c10d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/990784641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f7c1022a0 0x7f3f7c1026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/990784641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c100b50 0x7f3f7c100f30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 -- 172.21.15.64:0/990784641 >> 172.21.15.64:0/990784641 conn(0x7f3f7c0fc500 msgr2=0x7f3f7c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 -- 172.21.15.64:0/990784641 shutdown_connections 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.297+0000 7f3f83d9c640 1 -- 172.21.15.64:0/990784641 wait complete. 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 Processor -- start 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 -- start start 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f7c100b50 0x7f3f7c1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7c1022a0 0x7f3f7c1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c102c20 0x7f3f7c1a4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7c110cb0 con 0x7f3f7c102c20 2024-08-19T01:39:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7c110b30 con 0x7f3f7c1022a0 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f83d9c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7c110e30 con 0x7f3f7c100b50 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f82339640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c102c20 0x7f3f7c1a4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f82339640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c102c20 0x7f3f7c1a4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55058/0 (socket says 172.21.15.64:55058) 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.298+0000 7f3f82339640 1 -- 172.21.15.64:0/714343493 learned_addr learned my addr 172.21.15.64:0/714343493 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f81b38640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f7c100b50 0x7f3f7c1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f81337640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7c1022a0 0x7f3f7c1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f82339640 1 -- 172.21.15.64:0/714343493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f7c100b50 msgr2=0x7f3f7c1990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f82339640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f7c100b50 0x7f3f7c1990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f82339640 1 -- 172.21.15.64:0/714343493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7c1022a0 msgr2=0x7f3f7c1995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f82339640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7c1022a0 0x7f3f7c1995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f82339640 1 -- 172.21.15.64:0/714343493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7c1a4670 con 0x7f3f7c102c20 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f81337640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7c1022a0 0x7f3f7c1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f81b38640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f7c100b50 0x7f3f7c1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f82339640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c102c20 0x7f3f7c1a4130 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400eaa0 tx=0x7f3f7400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:46.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f72ffd640 1 -- 172.21.15.64:0/714343493 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7400cdb0 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f72ffd640 1 -- 172.21.15.64:0/714343493 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f74005020 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f7c1a4840 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f72ffd640 1 -- 172.21.15.64:0/714343493 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f740116b0 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.299+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f7c1a4d50 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.300+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7c100f30 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.302+0000 7f3f72ffd640 1 -- 172.21.15.64:0/714343493 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3f74005560 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.303+0000 7f3f72ffd640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f3f680a2d70 0x7f3f680a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.303+0000 7f3f72ffd640 1 -- 172.21.15.64:0/714343493 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3f740cfdb0 con 0x7f3f7c102c20 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.303+0000 7f3f81b38640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f3f680a2d70 0x7f3f680a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.304+0000 7f3f81b38640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f3f680a2d70 0x7f3f680a5230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005ab0 tx=0x7f3f78005a20 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:46.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.306+0000 7f3f72ffd640 1 -- 172.21.15.64:0/714343493 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f3f74016030 con 0x7f3f7c102c20 2024-08-19T01:39:46.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.40 MB/sec execute 198 sec latency 17472.835 ms 2024-08-19T01:39:46.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.519+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f7c108dc0 con 0x7f3f7c102c20 2024-08-19T01:39:46.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.519+0000 7f3f72ffd640 1 -- 172.21.15.64:0/714343493 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f3f7409c6a0 con 0x7f3f7c102c20 2024-08-19T01:39:46.521 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:39:46.521 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:39:46.521 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:39:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f3f680a2d70 msgr2=0x7f3f680a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f3f680a2d70 0x7f3f680a5230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005ab0 tx=0x7f3f78005a20 comp rx=0 tx=0).stop 2024-08-19T01:39:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c102c20 msgr2=0x7f3f7c1a4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c102c20 0x7f3f7c1a4130 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400eaa0 tx=0x7f3f7400ef60 comp rx=0 tx=0).stop 2024-08-19T01:39:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 shutdown_connections 2024-08-19T01:39:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f3f680a2d70 0x7f3f680a5230 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f7c102c20 0x7f3f7c1a4130 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7c1022a0 0x7f3f7c1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 --2- 172.21.15.64:0/714343493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f7c100b50 0x7f3f7c1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:46.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 >> 172.21.15.64:0/714343493 conn(0x7f3f7c0fc500 msgr2=0x7f3f7c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:46.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 shutdown_connections 2024-08-19T01:39:46.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:46.522+0000 7f3f83d9c640 1 -- 172.21.15.64:0/714343493 wait complete. 2024-08-19T01:39:47.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:46 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/714343493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:47.186 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:46 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/714343493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:47.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:46 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/714343493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:39:47.214 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id dfc10a11 --release --await' 2024-08-19T01:39:47.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id dfc10a11 --release --await 2024-08-19T01:39:47.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 559348 8.36 MB/sec execute 199 sec latency 18472.871 ms 2024-08-19T01:39:47.475 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:39:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:47 smithi064 ceph-mon[18515]: pgmap v199: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.8 MiB/s wr, 71 op/s 2024-08-19T01:39:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:47 smithi123 ceph-mon[21829]: pgmap v199: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.8 MiB/s wr, 71 op/s 2024-08-19T01:39:48.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.154+0000 7f45025f9640 1 -- 172.21.15.64:0/3960618934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc103390 msgr2=0x7f44fc10b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.154+0000 7f45025f9640 1 --2- 172.21.15.64:0/3960618934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc103390 0x7f44fc10b880 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009900 tx=0x7f44f002a480 comp rx=0 tx=0).stop 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.154+0000 7f45025f9640 1 -- 172.21.15.64:0/3960618934 shutdown_connections 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.154+0000 7f45025f9640 1 --2- 172.21.15.64:0/3960618934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44fc10bea0 0x7f44fc10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.154+0000 7f45025f9640 1 --2- 172.21.15.64:0/3960618934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc103390 0x7f44fc10b880 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.154+0000 7f45025f9640 1 --2- 172.21.15.64:0/3960618934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44fc102a70 0x7f44fc102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.154+0000 7f45025f9640 1 -- 172.21.15.64:0/3960618934 >> 172.21.15.64:0/3960618934 conn(0x7f44fc0fc4d0 msgr2=0x7f44fc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 -- 172.21.15.64:0/3960618934 shutdown_connections 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 -- 172.21.15.64:0/3960618934 wait complete. 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 Processor -- start 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 -- start start 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc102a70 0x7f44fc198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44fc103390 0x7f44fc1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44fc10bea0 0x7f44fc19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc111d60 con 0x7f44fc102a70 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc111be0 con 0x7f44fc103390 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.155+0000 7f45025f9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc111ee0 con 0x7f44fc10bea0 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc102a70 0x7f44fc198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc102a70 0x7f44fc198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55068/0 (socket says 172.21.15.64:55068) 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 -- 172.21.15.64:0/1671690679 learned_addr learned my addr 172.21.15.64:0/1671690679 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f4500b96640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44fc10bea0 0x7f44fc19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44fc103390 0x7f44fc1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 -- 172.21.15.64:0/1671690679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44fc10bea0 msgr2=0x7f44fc19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44fc10bea0 0x7f44fc19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 -- 172.21.15.64:0/1671690679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44fc103390 msgr2=0x7f44fc1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44fc103390 0x7f44fc1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 -- 172.21.15.64:0/1671690679 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc19d330 con 0x7f44fc102a70 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44fbfff640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc102a70 0x7f44fc198380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00b4a0 tx=0x7f44ec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44f97fa640 1 -- 172.21.15.64:0/1671690679 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec014070 con 0x7f44fc102a70 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44fc19d620 con 0x7f44fc102a70 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44f97fa640 1 -- 172.21.15.64:0/1671690679 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44ec010af0 con 0x7f44fc102a70 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f44f97fa640 1 -- 172.21.15.64:0/1671690679 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec017d10 con 0x7f44fc102a70 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.156+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44fc1a4f00 con 0x7f44fc102a70 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.159+0000 7f44f97fa640 1 -- 172.21.15.64:0/1671690679 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f44ec017070 con 0x7f44fc102a70 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.159+0000 7f45049f7740 1 Processor -- start 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.159+0000 7f45049f7740 1 -- start start 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44f97fa640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f44e40a2cf0 0x7f44e40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f45049f7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7eefc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f45049f7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f45049f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7f01370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f45049f7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5629a7e55c60 con 0x5629a7ef01b0 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f45049f7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5629a7dedb90 con 0x5629a7eef890 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f45049f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5629a7e51c20 con 0x5629a7efef80 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f4500b96640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f44e40a2cf0 0x7f44e40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55076/0 (socket says 172.21.15.64:55076) 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44f97fa640 1 -- 172.21.15.64:0/1671690679 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f44ec0ceb70 con 0x7f44fc102a70 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3248491108 learned_addr learned my addr 172.21.15.64:0/3248491108 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fbfff640 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7eefc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f4500b96640 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7f01370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3248491108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 msgr2=0x5629a7f01370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7f01370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3248491108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 msgr2=0x5629a7eefc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7eefc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3248491108 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629a7b65580 con 0x5629a7ef01b0 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f4500b96640 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7f01370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f4500b96640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f44e40a2cf0 0x7f44e40a51b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f44f40070b0 tx=0x7f44f400cb80 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.160+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efea40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f44f0005ab0 tx=0x7f44f00315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc5fc51575b0f63 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f44e27fc640 1 -- 172.21.15.64:0/3248491108 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f003f070 con 0x5629a7ef01b0 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f44e27fc640 1 -- 172.21.15.64:0/3248491108 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f00319a0 con 0x5629a7ef01b0 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f44e27fc640 1 -- 172.21.15.64:0/3248491108 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0035070 con 0x5629a7ef01b0 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 -- 172.21.15.64:0/3248491108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 msgr2=0x5629a7efea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efea40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f44f0005ab0 tx=0x7f44f00315b0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 -- 172.21.15.64:0/3248491108 shutdown_connections 2024-08-19T01:39:48.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7f01370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efea40 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 --2- 172.21.15.64:0/3248491108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7eefc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 -- 172.21.15.64:0/3248491108 >> 172.21.15.64:0/3248491108 conn(0x5629a7eebf20 msgr2=0x5629a7eec300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 -- 172.21.15.64:0/3248491108 shutdown_connections 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.161+0000 7f45049f7740 1 -- 172.21.15.64:0/3248491108 wait complete. 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 -- start start 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 Processor -- start 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7efbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7efc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5629a7f01b50 con 0x5629a7ef01b0 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5629a7df37c0 con 0x5629a7eef890 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5629a7f01cd0 con 0x5629a7efef80 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f45049f7740 10 client.0 _ll_register_callbacks cb 0x5629a7dfcf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55086/0 (socket says 172.21.15.64:55086) 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3714214652 learned_addr learned my addr 172.21.15.64:0/3714214652 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.162+0000 7f44fbfff640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7efbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f4500b96640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7efc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3714214652 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 msgr2=0x5629a7efc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7efc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 msgr2=0x5629a7efbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7efbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44fb7fe640 1 -- 172.21.15.64:0/3714214652 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629a7c2f960 con 0x5629a7ef01b0 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44fbfff640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7efbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efc430 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f44f0006010 tx=0x7f44f0004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a72b20596585b6 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44db7fe640 1 -- 172.21.15.64:0/3714214652 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0035030 con 0x5629a7ef01b0 2024-08-19T01:39:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44db7fe640 1 -- 172.21.15.64:0/3714214652 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f44f0004050 con 0x5629a7ef01b0 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44f00319a0 con 0x5629a7ef01b0 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44db7fe640 1 -- 172.21.15.64:0/3714214652 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f44f0055050 con 0x5629a7ef01b0 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f45049f7740 10 client.15753 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5629a7f0dc50 con 0x5629a7ef01b0 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.163+0000 7f44db7fe640 1 -- 172.21.15.64:0/3714214652 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f00319a0 con 0x5629a7ef01b0 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f44db7fe640 1 -- 172.21.15.64:0/3714214652 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f44f003b430 con 0x5629a7ef01b0 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f45049f7740 10 client.15753 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f45049f7740 10 client.15753 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f45049f7740 20 client.15753 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f45049f7740 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f0efa0 0x5629a7f11390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f45049f7740 4 client.15753 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629a7e631d0 con 0x5629a7f0efa0 2024-08-19T01:39:48.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f44fbfff640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f0efa0 0x5629a7f11390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f44db7fe640 10 client.15753 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.164+0000 7f44fbfff640 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f0efa0 0x5629a7f11390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fc5bda45f3567e3 server_cookie=edc9f32d93d459b3 in_seq=0 out_seq=0 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f44db7fe640 1 -- 172.21.15.64:0/3714214652 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5629a7e631d0 con 0x5629a7f0efa0 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f44db7fe640 10 client.15753 handle_command_reply: tid=0 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 client.15753 shutdown 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 msgr2=0x5629a7efc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efc430 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f44f0006010 tx=0x7f44f0004a50 comp rx=0 tx=0).stop 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 shutdown_connections 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7efef80 0x5629a7efc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7efc430 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f0efa0 0x5629a7f11390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 --2- 172.21.15.64:0/3714214652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7eef890 0x5629a7efbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f44db7fe640 0 client.15753 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:48.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.165+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 >> 172.21.15.64:0/3714214652 conn(0x5629a7eebf20 msgr2=0x5629a7ef1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:48.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.166+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 shutdown_connections 2024-08-19T01:39:48.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.166+0000 7f45049f7740 1 -- 172.21.15.64:0/3714214652 wait complete. 2024-08-19T01:39:48.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.166+0000 7f45049f7740 20 client.15753 trim_cache size 0 max 16384 2024-08-19T01:39:48.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 Processor -- start 2024-08-19T01:39:48.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 -- start start 2024-08-19T01:39:48.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7ef9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7ef01b0 0x5629a7ef9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7efef80 0x5629a7ef1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1af500 con 0x5629a7efef80 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1acdd0 con 0x5629a7ef01b0 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f45049f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1aac20 con 0x5629a7eef890 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7efef80 0x5629a7ef1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7efef80 0x5629a7ef1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55092/0 (socket says 172.21.15.64:55092) 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 -- 172.21.15.64:0/3817293241 learned_addr learned my addr 172.21.15.64:0/3817293241 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f44fbfff640 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7ef9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7ef01b0 0x5629a7ef9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 -- 172.21.15.64:0/3817293241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 msgr2=0x5629a7ef9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7ef9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 -- 172.21.15.64:0/3817293241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7ef01b0 msgr2=0x5629a7ef9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7ef01b0 0x5629a7ef9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.183+0000 7f4500b96640 1 -- 172.21.15.64:0/3817293241 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629a7eeffe0 con 0x5629a7efef80 2024-08-19T01:39:48.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f4500b96640 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7efef80 0x5629a7ef1460 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f44f400ce80 tx=0x7f44f4009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212664b8baa42c22 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:48.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f44e1ffb640 1 -- 172.21.15.64:0/3817293241 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f401c640 con 0x5629a7efef80 2024-08-19T01:39:48.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f44e1ffb640 1 -- 172.21.15.64:0/3817293241 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f4016040 con 0x5629a7efef80 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f44e1ffb640 1 -- 172.21.15.64:0/3817293241 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f401aa30 con 0x5629a7efef80 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f45049f7740 1 -- 172.21.15.64:0/3817293241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7efef80 msgr2=0x5629a7ef1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f45049f7740 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7efef80 0x5629a7ef1460 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f44f400ce80 tx=0x7f44f4009470 comp rx=0 tx=0).stop 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f45049f7740 1 -- 172.21.15.64:0/3817293241 shutdown_connections 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f45049f7740 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7efef80 0x5629a7ef1460 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f45049f7740 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7ef01b0 0x5629a7ef9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.184+0000 7f45049f7740 1 --2- 172.21.15.64:0/3817293241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7ef9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 -- 172.21.15.64:0/3817293241 >> 172.21.15.64:0/3817293241 conn(0x5629a7eebf20 msgr2=0x5629a7ef71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 -- 172.21.15.64:0/3817293241 shutdown_connections 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 -- 172.21.15.64:0/3817293241 wait complete. 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 -- start start 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 Processor -- start 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7f12100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7f12640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7efef80 0x5629a7f12b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1ab290 con 0x5629a7ef01b0 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1ac700 con 0x5629a7efef80 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.185+0000 7f45049f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1ab960 con 0x5629a7eef890 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f45049f7740 10 client.0 _ll_register_callbacks cb 0x7f44f0005370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7f12640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7f12640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55108/0 (socket says 172.21.15.64:55108) 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 -- 172.21.15.64:0/550413668 learned_addr learned my addr 172.21.15.64:0/550413668 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f4500b96640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7efef80 0x5629a7f12b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fbfff640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7f12100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 -- 172.21.15.64:0/550413668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 msgr2=0x5629a7f12100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7f12100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 -- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7efef80 msgr2=0x5629a7f12b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7efef80 0x5629a7f12b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 -- 172.21.15.64:0/550413668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629a7f788e0 con 0x5629a7ef01b0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f4500b96640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7efef80 0x5629a7f12b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44fb7fe640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7f12640 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x5629a7efd530 tx=0x7f44f0009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb39f00ca4ae107 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44db7fe640 1 -- 172.21.15.64:0/550413668 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0035070 con 0x5629a7ef01b0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44db7fe640 1 -- 172.21.15.64:0/550413668 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f44f00315b0 con 0x5629a7ef01b0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44f4016040 con 0x5629a7ef01b0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f44db7fe640 1 -- 172.21.15.64:0/550413668 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f44f003f040 con 0x5629a7ef01b0 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f45049f7740 10 client.15759 fetch_fsmap learned FSMap version 29 2024-08-19T01:39:48.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.186+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5629a7f79bd0 con 0x5629a7ef01b0 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f44db7fe640 1 -- 172.21.15.64:0/550413668 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f004c070 con 0x5629a7ef01b0 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f44db7fe640 1 -- 172.21.15.64:0/550413668 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f44f0061030 con 0x5629a7ef01b0 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f45049f7740 10 client.15759 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f45049f7740 10 client.15759 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f45049f7740 20 client.15759 populate_metadata read hostname 'smithi064' 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f45049f7740 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f7a3d0 0x5629a7f7c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f45049f7740 4 client.15759 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "dfc10a11", "release": true, "await": true}] 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "dfc10a11", "release": true, "await": true}) -- 0x7f44f4016040 con 0x5629a7f7a3d0 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f44fbfff640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f7a3d0 0x5629a7f7c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f44fbfff640 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f7a3d0 0x5629a7f7c7f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddcc4cf328724ce3 server_cookie=f4d99a4a20de44ae in_seq=0 out_seq=0 2024-08-19T01:39:48.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.187+0000 7f44db7fe640 10 client.15759 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f44db7fe640 1 -- 172.21.15.64:0/550413668 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f44f4016040 con 0x5629a7f7a3d0 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f44db7fe640 10 client.15759 handle_command_reply: tid=0 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 client.15759 shutdown 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 msgr2=0x5629a7f12640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7f12640 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x5629a7efd530 tx=0x7f44f0009680 comp rx=0 tx=0).stop 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 shutdown_connections 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5629a7f7a3d0 0x5629a7f7c7f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5629a7efef80 0x5629a7f12b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629a7ef01b0 0x5629a7f12640 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 --2- 172.21.15.64:0/550413668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5629a7eef890 0x5629a7f12100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 >> 172.21.15.64:0/550413668 conn(0x5629a7eebf20 msgr2=0x5629a7efd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f44db7fe640 0 client.15759 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:39:48.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 shutdown_connections 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 1 -- 172.21.15.64:0/550413668 wait complete. 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.197+0000 7f45049f7740 20 client.15759 trim_cache size 0 max 16384 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 24, 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: "dfc10a11": { 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 24, 2024-08-19T01:39:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 139.2, 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:39:48.202 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:39:48.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f44e40a2cf0 msgr2=0x7f44e40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f44e40a2cf0 0x7f44e40a51b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f44f40070b0 tx=0x7f44f400cb80 comp rx=0 tx=0).stop 2024-08-19T01:39:48.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc102a70 msgr2=0x7f44fc198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc102a70 0x7f44fc198380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00b4a0 tx=0x7f44ec00b960 comp rx=0 tx=0).stop 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 shutdown_connections 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f44e40a2cf0 0x7f44e40a51b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44fc10bea0 0x7f44fc19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44fc103390 0x7f44fc1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 --2- 172.21.15.64:0/1671690679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44fc102a70 0x7f44fc198380 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 >> 172.21.15.64:0/1671690679 conn(0x7f44fc0fc4d0 msgr2=0x7f44fc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 shutdown_connections 2024-08-19T01:39:48.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:39:48.198+0000 7f45025f9640 1 -- 172.21.15.64:0/1671690679 wait complete. 2024-08-19T01:39:48.203 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:47 smithi161 ceph-mon[21757]: pgmap v199: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 1.8 MiB/s wr, 71 op/s 2024-08-19T01:39:48.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 560040 8.33 MB/sec execute 200 sec latency 19277.695 ms 2024-08-19T01:39:48.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:48 smithi064 ceph-mon[18515]: pgmap v200: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 973 KiB/s wr, 37 op/s 2024-08-19T01:39:48.952 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'dfc10a11', total seconds elapsed: 22.6 2024-08-19T01:39:49.066 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:48 smithi123 ceph-mon[21829]: pgmap v200: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 973 KiB/s wr, 37 op/s 2024-08-19T01:39:49.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:48 smithi161 ceph-mon[21757]: pgmap v200: 129 pgs: 129 active+clean; 350 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 973 KiB/s wr, 37 op/s 2024-08-19T01:39:49.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 562361 8.36 MB/sec execute 201 sec latency 10.976 ms 2024-08-19T01:39:50.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 565123 8.40 MB/sec execute 202 sec latency 21.168 ms 2024-08-19T01:39:51.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:50 smithi064 ceph-mon[18515]: pgmap v201: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.3 MiB/s wr, 65 op/s 2024-08-19T01:39:51.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 567263 8.40 MB/sec execute 203 sec latency 21.838 ms 2024-08-19T01:39:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:50 smithi123 ceph-mon[21829]: pgmap v201: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.3 MiB/s wr, 65 op/s 2024-08-19T01:39:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:50 smithi161 ceph-mon[21757]: pgmap v201: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.3 MiB/s wr, 65 op/s 2024-08-19T01:39:52.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 569531 8.42 MB/sec execute 204 sec latency 11.678 ms 2024-08-19T01:39:53.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:52 smithi064 ceph-mon[18515]: pgmap v202: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 591 KiB/s wr, 39 op/s 2024-08-19T01:39:53.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 572165 8.46 MB/sec execute 205 sec latency 18.641 ms 2024-08-19T01:39:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:52 smithi123 ceph-mon[21829]: pgmap v202: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 591 KiB/s wr, 39 op/s 2024-08-19T01:39:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:52 smithi161 ceph-mon[21757]: pgmap v202: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 591 KiB/s wr, 39 op/s 2024-08-19T01:39:54.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 574271 8.46 MB/sec execute 206 sec latency 35.459 ms 2024-08-19T01:39:55.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 576413 8.48 MB/sec execute 207 sec latency 12.811 ms 2024-08-19T01:39:55.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:55 smithi161 ceph-mon[21757]: pgmap v203: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 591 KiB/s wr, 39 op/s 2024-08-19T01:39:55.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:55 smithi064 ceph-mon[18515]: pgmap v203: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 591 KiB/s wr, 39 op/s 2024-08-19T01:39:56.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:55 smithi123 ceph-mon[21829]: pgmap v203: 129 pgs: 129 active+clean; 354 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 591 KiB/s wr, 39 op/s 2024-08-19T01:39:56.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 579233 8.52 MB/sec execute 208 sec latency 20.299 ms 2024-08-19T01:39:57.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 581406 8.52 MB/sec execute 209 sec latency 16.476 ms 2024-08-19T01:39:57.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:57 smithi161 ceph-mon[21757]: pgmap v204: 129 pgs: 129 active+clean; 363 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2024-08-19T01:39:57.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:57 smithi064 ceph-mon[18515]: pgmap v204: 129 pgs: 129 active+clean; 363 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2024-08-19T01:39:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:57 smithi123 ceph-mon[21829]: pgmap v204: 129 pgs: 129 active+clean; 363 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2024-08-19T01:39:58.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 583791 8.55 MB/sec execute 210 sec latency 17.741 ms 2024-08-19T01:39:59.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 586490 8.59 MB/sec execute 211 sec latency 24.099 ms 2024-08-19T01:39:59.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:39:59 smithi161 ceph-mon[21757]: pgmap v205: 129 pgs: 129 active+clean; 363 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-08-19T01:39:59.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:39:59 smithi064 ceph-mon[18515]: pgmap v205: 129 pgs: 129 active+clean; 363 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-08-19T01:40:00.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:39:59 smithi123 ceph-mon[21829]: pgmap v205: 129 pgs: 129 active+clean; 363 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-08-19T01:40:00.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 588850 8.60 MB/sec execute 212 sec latency 12.305 ms 2024-08-19T01:40:00.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:00 smithi161 ceph-mon[21757]: overall HEALTH_OK 2024-08-19T01:40:00.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:00 smithi064 ceph-mon[18515]: overall HEALTH_OK 2024-08-19T01:40:01.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:00 smithi123 ceph-mon[21829]: overall HEALTH_OK 2024-08-19T01:40:01.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 589707 8.59 MB/sec execute 213 sec latency 85.895 ms 2024-08-19T01:40:01.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:01 smithi161 ceph-mon[21757]: pgmap v206: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-08-19T01:40:01.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:01 smithi064 ceph-mon[18515]: pgmap v206: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-08-19T01:40:02.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:01 smithi123 ceph-mon[21829]: pgmap v206: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-08-19T01:40:02.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 591396 8.58 MB/sec execute 214 sec latency 16.290 ms 2024-08-19T01:40:03.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 593784 8.60 MB/sec execute 215 sec latency 14.407 ms 2024-08-19T01:40:03.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:03 smithi064 ceph-mon[18515]: pgmap v207: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-08-19T01:40:04.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:03 smithi123 ceph-mon[21829]: pgmap v207: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-08-19T01:40:04.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:03 smithi161 ceph-mon[21757]: pgmap v207: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-08-19T01:40:04.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 596527 8.63 MB/sec execute 216 sec latency 12.102 ms 2024-08-19T01:40:05.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 598270 8.64 MB/sec execute 217 sec latency 27.163 ms 2024-08-19T01:40:05.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:05 smithi064 ceph-mon[18515]: pgmap v208: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-08-19T01:40:06.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:05 smithi123 ceph-mon[21829]: pgmap v208: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-08-19T01:40:06.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:05 smithi161 ceph-mon[21757]: pgmap v208: 129 pgs: 129 active+clean; 374 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 161 op/s 2024-08-19T01:40:06.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 600762 8.66 MB/sec execute 218 sec latency 14.012 ms 2024-08-19T01:40:07.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 602825 8.66 MB/sec execute 219 sec latency 104.975 ms 2024-08-19T01:40:07.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:07 smithi064 ceph-mon[18515]: pgmap v209: 129 pgs: 129 active+clean; 382 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 282 op/s 2024-08-19T01:40:07.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:07 smithi064 ceph-mon[18515]: osdmap e95: 12 total, 12 up, 12 in 2024-08-19T01:40:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:07 smithi123 ceph-mon[21829]: pgmap v209: 129 pgs: 129 active+clean; 382 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 282 op/s 2024-08-19T01:40:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:07 smithi123 ceph-mon[21829]: osdmap e95: 12 total, 12 up, 12 in 2024-08-19T01:40:08.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:07 smithi161 ceph-mon[21757]: pgmap v209: 129 pgs: 129 active+clean; 382 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.4 MiB/s wr, 282 op/s 2024-08-19T01:40:08.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:07 smithi161 ceph-mon[21757]: osdmap e95: 12 total, 12 up, 12 in 2024-08-19T01:40:08.393 INFO:tasks.workunit.client.0.smithi064.stdout: 1 604967 8.68 MB/sec execute 220 sec latency 18.886 ms 2024-08-19T01:40:09.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 607637 8.72 MB/sec execute 221 sec latency 25.957 ms 2024-08-19T01:40:09.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:09 smithi064 ceph-mon[18515]: pgmap v211: 129 pgs: 129 active+clean; 382 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 246 op/s 2024-08-19T01:40:10.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:09 smithi123 ceph-mon[21829]: pgmap v211: 129 pgs: 129 active+clean; 382 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 246 op/s 2024-08-19T01:40:10.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:09 smithi161 ceph-mon[21757]: pgmap v211: 129 pgs: 129 active+clean; 382 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 246 op/s 2024-08-19T01:40:10.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 609752 8.71 MB/sec execute 222 sec latency 18.560 ms 2024-08-19T01:40:11.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 612036 8.73 MB/sec execute 223 sec latency 10.906 ms 2024-08-19T01:40:11.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:11 smithi064 ceph-mon[18515]: pgmap v212: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 239 op/s 2024-08-19T01:40:12.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:11 smithi123 ceph-mon[21829]: pgmap v212: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 239 op/s 2024-08-19T01:40:12.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:11 smithi161 ceph-mon[21757]: pgmap v212: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 239 op/s 2024-08-19T01:40:12.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 614805 8.77 MB/sec execute 224 sec latency 27.062 ms 2024-08-19T01:40:13.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 616892 8.76 MB/sec execute 225 sec latency 23.993 ms 2024-08-19T01:40:13.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:13 smithi064 ceph-mon[18515]: pgmap v213: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 239 op/s 2024-08-19T01:40:13.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:13 smithi064 ceph-mon[18515]: osdmap e96: 12 total, 12 up, 12 in 2024-08-19T01:40:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:13 smithi123 ceph-mon[21829]: pgmap v213: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 239 op/s 2024-08-19T01:40:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:13 smithi123 ceph-mon[21829]: osdmap e96: 12 total, 12 up, 12 in 2024-08-19T01:40:14.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:13 smithi161 ceph-mon[21757]: pgmap v213: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 239 op/s 2024-08-19T01:40:14.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:13 smithi161 ceph-mon[21757]: osdmap e96: 12 total, 12 up, 12 in 2024-08-19T01:40:14.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 619204 8.79 MB/sec execute 226 sec latency 12.421 ms 2024-08-19T01:40:15.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 621838 8.82 MB/sec execute 227 sec latency 25.373 ms 2024-08-19T01:40:15.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:15 smithi064 ceph-mon[18515]: pgmap v215: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2024-08-19T01:40:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:15 smithi123 ceph-mon[21829]: pgmap v215: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2024-08-19T01:40:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:15 smithi161 ceph-mon[21757]: pgmap v215: 129 pgs: 129 active+clean; 393 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.5 MiB/s wr, 117 op/s 2024-08-19T01:40:16.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 623969 8.82 MB/sec execute 228 sec latency 13.271 ms 2024-08-19T01:40:17.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 626145 8.84 MB/sec execute 229 sec latency 14.408 ms 2024-08-19T01:40:17.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:17 smithi064 ceph-mon[18515]: pgmap v216: 129 pgs: 129 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.7 MiB/s wr, 208 op/s 2024-08-19T01:40:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:17 smithi123 ceph-mon[21829]: pgmap v216: 129 pgs: 129 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.7 MiB/s wr, 208 op/s 2024-08-19T01:40:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:17 smithi161 ceph-mon[21757]: pgmap v216: 129 pgs: 129 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.7 MiB/s wr, 208 op/s 2024-08-19T01:40:18.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 628873 8.87 MB/sec execute 230 sec latency 24.748 ms 2024-08-19T01:40:19.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 630977 8.87 MB/sec execute 231 sec latency 12.929 ms 2024-08-19T01:40:19.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:19 smithi064 ceph-mon[18515]: pgmap v217: 129 pgs: 129 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-08-19T01:40:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:19 smithi123 ceph-mon[21829]: pgmap v217: 129 pgs: 129 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-08-19T01:40:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:19 smithi161 ceph-mon[21757]: pgmap v217: 129 pgs: 129 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-08-19T01:40:20.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 633225 8.89 MB/sec execute 232 sec latency 17.528 ms 2024-08-19T01:40:21.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 635932 8.92 MB/sec execute 233 sec latency 28.359 ms 2024-08-19T01:40:21.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:21 smithi064 ceph-mon[18515]: pgmap v218: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-08-19T01:40:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:21 smithi123 ceph-mon[21829]: pgmap v218: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-08-19T01:40:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:21 smithi161 ceph-mon[21757]: pgmap v218: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-08-19T01:40:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 637284 8.91 MB/sec execute 234 sec latency 55.698 ms 2024-08-19T01:40:22.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:40:22.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:40:22.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:40:22.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:22 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:40:23.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:40:23.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:40:23.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:40:23.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:22 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:40:23.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:40:23.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:40:23.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:40:23.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:22 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:40:23.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 639850 8.93 MB/sec execute 235 sec latency 12.615 ms 2024-08-19T01:40:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:23 smithi064 ceph-mon[18515]: pgmap v219: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-08-19T01:40:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:23 smithi123 ceph-mon[21829]: pgmap v219: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-08-19T01:40:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:23 smithi161 ceph-mon[21757]: pgmap v219: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-08-19T01:40:24.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 642444 8.95 MB/sec execute 236 sec latency 14.104 ms 2024-08-19T01:40:25.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 644362 8.96 MB/sec execute 237 sec latency 19.955 ms 2024-08-19T01:40:25.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:25 smithi064 ceph-mon[18515]: pgmap v220: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 189 op/s 2024-08-19T01:40:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:25 smithi123 ceph-mon[21829]: pgmap v220: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 189 op/s 2024-08-19T01:40:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:25 smithi161 ceph-mon[21757]: pgmap v220: 129 pgs: 129 active+clean; 415 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 189 op/s 2024-08-19T01:40:26.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 646913 8.98 MB/sec execute 238 sec latency 11.981 ms 2024-08-19T01:40:27.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 649127 8.99 MB/sec execute 239 sec latency 117.041 ms 2024-08-19T01:40:27.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:27 smithi064 ceph-mon[18515]: pgmap v221: 129 pgs: 129 active+clean; 425 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.8 MiB/s wr, 260 op/s 2024-08-19T01:40:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:27 smithi123 ceph-mon[21829]: pgmap v221: 129 pgs: 129 active+clean; 425 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.8 MiB/s wr, 260 op/s 2024-08-19T01:40:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:27 smithi161 ceph-mon[21757]: pgmap v221: 129 pgs: 129 active+clean; 425 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.8 MiB/s wr, 260 op/s 2024-08-19T01:40:28.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 651045 8.99 MB/sec execute 240 sec latency 18.020 ms 2024-08-19T01:40:29.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 653766 9.02 MB/sec execute 241 sec latency 29.990 ms 2024-08-19T01:40:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:29 smithi123 ceph-mon[21829]: pgmap v222: 129 pgs: 129 active+clean; 425 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:40:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:29 smithi161 ceph-mon[21757]: pgmap v222: 129 pgs: 129 active+clean; 425 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:40:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:29 smithi064 ceph-mon[18515]: pgmap v222: 129 pgs: 129 active+clean; 425 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:40:30.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 655997 9.02 MB/sec execute 242 sec latency 14.590 ms 2024-08-19T01:40:31.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 658385 9.05 MB/sec execute 243 sec latency 18.658 ms 2024-08-19T01:40:32.000 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:31 smithi161 ceph-mon[21757]: pgmap v223: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.6 MiB/s wr, 287 op/s 2024-08-19T01:40:32.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:31 smithi064 ceph-mon[18515]: pgmap v223: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.6 MiB/s wr, 287 op/s 2024-08-19T01:40:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:31 smithi123 ceph-mon[21829]: pgmap v223: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.6 MiB/s wr, 287 op/s 2024-08-19T01:40:32.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 660763 9.07 MB/sec execute 244 sec latency 58.087 ms 2024-08-19T01:40:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 662798 9.06 MB/sec execute 245 sec latency 39.135 ms 2024-08-19T01:40:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:33 smithi123 ceph-mon[21829]: pgmap v224: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 207 op/s 2024-08-19T01:40:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:33 smithi161 ceph-mon[21757]: pgmap v224: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 207 op/s 2024-08-19T01:40:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:33 smithi064 ceph-mon[18515]: pgmap v224: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 207 op/s 2024-08-19T01:40:34.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 664976 9.08 MB/sec execute 246 sec latency 11.629 ms 2024-08-19T01:40:35.394 INFO:tasks.workunit.client.0.smithi064.stdout: 1 667708 9.11 MB/sec execute 247 sec latency 21.838 ms 2024-08-19T01:40:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:35 smithi123 ceph-mon[21829]: pgmap v225: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 207 op/s 2024-08-19T01:40:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:35 smithi161 ceph-mon[21757]: pgmap v225: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 207 op/s 2024-08-19T01:40:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:35 smithi064 ceph-mon[18515]: pgmap v225: 129 pgs: 129 active+clean; 436 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 207 op/s 2024-08-19T01:40:36.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 669448 9.10 MB/sec execute 248 sec latency 17.002 ms 2024-08-19T01:40:37.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 671695 9.12 MB/sec execute 249 sec latency 55.189 ms 2024-08-19T01:40:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:37 smithi123 ceph-mon[21829]: pgmap v226: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.4 MiB/s wr, 310 op/s 2024-08-19T01:40:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:37 smithi161 ceph-mon[21757]: pgmap v226: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.4 MiB/s wr, 310 op/s 2024-08-19T01:40:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:37 smithi064 ceph-mon[18515]: pgmap v226: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.4 MiB/s wr, 310 op/s 2024-08-19T01:40:38.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 673777 9.11 MB/sec execute 250 sec latency 64.899 ms 2024-08-19T01:40:39.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 675954 9.13 MB/sec execute 251 sec latency 17.400 ms 2024-08-19T01:40:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:39 smithi123 ceph-mon[21829]: pgmap v227: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 211 op/s 2024-08-19T01:40:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:39 smithi161 ceph-mon[21757]: pgmap v227: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 211 op/s 2024-08-19T01:40:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:39 smithi064 ceph-mon[18515]: pgmap v227: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 211 op/s 2024-08-19T01:40:40.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 678680 9.16 MB/sec execute 252 sec latency 28.523 ms 2024-08-19T01:40:41.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 680800 9.16 MB/sec execute 253 sec latency 15.034 ms 2024-08-19T01:40:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:41 smithi123 ceph-mon[21829]: pgmap v228: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 311 op/s 2024-08-19T01:40:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:41 smithi161 ceph-mon[21757]: pgmap v228: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 311 op/s 2024-08-19T01:40:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:41 smithi064 ceph-mon[18515]: pgmap v228: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 311 op/s 2024-08-19T01:40:42.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 682643 9.17 MB/sec execute 254 sec latency 96.203 ms 2024-08-19T01:40:43.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 685151 9.19 MB/sec execute 255 sec latency 45.258 ms 2024-08-19T01:40:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:43 smithi123 ceph-mon[21829]: pgmap v229: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 202 op/s 2024-08-19T01:40:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:43 smithi161 ceph-mon[21757]: pgmap v229: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 202 op/s 2024-08-19T01:40:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:43 smithi064 ceph-mon[18515]: pgmap v229: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 202 op/s 2024-08-19T01:40:44.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 686994 9.19 MB/sec execute 256 sec latency 14.606 ms 2024-08-19T01:40:45.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 689568 9.21 MB/sec execute 257 sec latency 12.130 ms 2024-08-19T01:40:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:45 smithi123 ceph-mon[21829]: pgmap v230: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 202 op/s 2024-08-19T01:40:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:45 smithi161 ceph-mon[21757]: pgmap v230: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 202 op/s 2024-08-19T01:40:46.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:45 smithi064 ceph-mon[18515]: pgmap v230: 129 pgs: 129 active+clean; 430 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 202 op/s 2024-08-19T01:40:46.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 692312 9.23 MB/sec execute 258 sec latency 23.901 ms 2024-08-19T01:40:47.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 693997 9.23 MB/sec execute 259 sec latency 62.362 ms 2024-08-19T01:40:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:47 smithi123 ceph-mon[21829]: pgmap v231: 129 pgs: 129 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.4 MiB/s wr, 313 op/s 2024-08-19T01:40:48.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:47 smithi161 ceph-mon[21757]: pgmap v231: 129 pgs: 129 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.4 MiB/s wr, 313 op/s 2024-08-19T01:40:48.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:47 smithi064 ceph-mon[18515]: pgmap v231: 129 pgs: 129 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.4 MiB/s wr, 313 op/s 2024-08-19T01:40:48.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 696361 9.25 MB/sec execute 260 sec latency 22.359 ms 2024-08-19T01:40:49.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 698635 9.24 MB/sec execute 261 sec latency 13.881 ms 2024-08-19T01:40:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:49 smithi123 ceph-mon[21829]: pgmap v232: 129 pgs: 129 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 209 op/s 2024-08-19T01:40:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:49 smithi161 ceph-mon[21757]: pgmap v232: 129 pgs: 129 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 209 op/s 2024-08-19T01:40:50.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:49 smithi064 ceph-mon[18515]: pgmap v232: 129 pgs: 129 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 209 op/s 2024-08-19T01:40:50.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 700863 9.26 MB/sec execute 262 sec latency 11.943 ms 2024-08-19T01:40:51.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 703538 9.29 MB/sec execute 263 sec latency 19.222 ms 2024-08-19T01:40:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:51 smithi123 ceph-mon[21829]: pgmap v233: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.5 MiB/s wr, 316 op/s 2024-08-19T01:40:52.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:51 smithi161 ceph-mon[21757]: pgmap v233: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.5 MiB/s wr, 316 op/s 2024-08-19T01:40:52.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:51 smithi064 ceph-mon[18515]: pgmap v233: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.5 MiB/s wr, 316 op/s 2024-08-19T01:40:52.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 705271 9.27 MB/sec execute 264 sec latency 82.315 ms 2024-08-19T01:40:53.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 707413 9.29 MB/sec execute 265 sec latency 21.810 ms 2024-08-19T01:40:54.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:53 smithi123 ceph-mon[21829]: pgmap v234: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 216 op/s 2024-08-19T01:40:54.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:53 smithi161 ceph-mon[21757]: pgmap v234: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 216 op/s 2024-08-19T01:40:54.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:53 smithi064 ceph-mon[18515]: pgmap v234: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 216 op/s 2024-08-19T01:40:54.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 710225 9.32 MB/sec execute 266 sec latency 19.730 ms 2024-08-19T01:40:55.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 711873 9.31 MB/sec execute 267 sec latency 17.226 ms 2024-08-19T01:40:56.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:55 smithi123 ceph-mon[21829]: pgmap v235: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 216 op/s 2024-08-19T01:40:56.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:55 smithi161 ceph-mon[21757]: pgmap v235: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 216 op/s 2024-08-19T01:40:56.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:55 smithi064 ceph-mon[18515]: pgmap v235: 129 pgs: 129 active+clean; 449 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 216 op/s 2024-08-19T01:40:56.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 714384 9.33 MB/sec execute 268 sec latency 11.362 ms 2024-08-19T01:40:57.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 716752 9.34 MB/sec execute 269 sec latency 43.615 ms 2024-08-19T01:40:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:57 smithi123 ceph-mon[21829]: pgmap v236: 129 pgs: 129 active+clean; 459 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.6 MiB/s wr, 318 op/s 2024-08-19T01:40:58.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:57 smithi161 ceph-mon[21757]: pgmap v236: 129 pgs: 129 active+clean; 459 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.6 MiB/s wr, 318 op/s 2024-08-19T01:40:58.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:57 smithi064 ceph-mon[18515]: pgmap v236: 129 pgs: 129 active+clean; 459 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.6 MiB/s wr, 318 op/s 2024-08-19T01:40:58.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 718293 9.33 MB/sec execute 270 sec latency 80.529 ms 2024-08-19T01:40:59.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 721100 9.37 MB/sec execute 271 sec latency 13.253 ms 2024-08-19T01:41:00.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:40:59 smithi123 ceph-mon[21829]: pgmap v237: 129 pgs: 129 active+clean; 459 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 208 op/s 2024-08-19T01:41:00.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:40:59 smithi161 ceph-mon[21757]: pgmap v237: 129 pgs: 129 active+clean; 459 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 208 op/s 2024-08-19T01:41:00.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:40:59 smithi064 ceph-mon[18515]: pgmap v237: 129 pgs: 129 active+clean; 459 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 208 op/s 2024-08-19T01:41:00.268 INFO:tasks.quiescer.fs.[cephfs]:Total duration (47.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-19T01:41:00.268 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.75 2024-08-19T01:41:00.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:41:00.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 723121 9.35 MB/sec execute 272 sec latency 20.197 ms 2024-08-19T01:41:00.568 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:01.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 -- 172.21.15.64:0/3430132299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc1078b0 msgr2=0x7f1ddc107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:01.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 --2- 172.21.15.64:0/3430132299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc1078b0 0x7f1ddc107c90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009930 tx=0x7f1dcc02a420 comp rx=0 tx=0).stop 2024-08-19T01:41:01.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 -- 172.21.15.64:0/3430132299 shutdown_connections 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 --2- 172.21.15.64:0/3430132299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ddc102250 0x7f1ddc10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 --2- 172.21.15.64:0/3430132299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ddc1018b0 0x7f1ddc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 --2- 172.21.15.64:0/3430132299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc1078b0 0x7f1ddc107c90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 -- 172.21.15.64:0/3430132299 >> 172.21.15.64:0/3430132299 conn(0x7f1ddc0fd5b0 msgr2=0x7f1ddc0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 -- 172.21.15.64:0/3430132299 shutdown_connections 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.180+0000 7f1de17a9640 1 -- 172.21.15.64:0/3430132299 wait complete. 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 Processor -- start 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 -- start start 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ddc1018b0 0x7f1ddc198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc102250 0x7f1ddc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ddc1078b0 0x7f1ddc19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc111cb0 con 0x7f1ddc102250 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc111b30 con 0x7f1ddc1078b0 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1de17a9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1ddc111e30 con 0x7f1ddc1018b0 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1dda7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc102250 0x7f1ddc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1ddaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ddc1018b0 0x7f1ddc198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1dda7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc102250 0x7f1ddc1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33136/0 (socket says 172.21.15.64:33136) 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.181+0000 7f1dda7fc640 1 -- 172.21.15.64:0/1483200572 learned_addr learned my addr 172.21.15.64:0/1483200572 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1ddb7fe640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ddc1078b0 0x7f1ddc19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1ddaffd640 1 -- 172.21.15.64:0/1483200572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ddc1078b0 msgr2=0x7f1ddc19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1ddaffd640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ddc1078b0 0x7f1ddc19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1ddaffd640 1 -- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc102250 msgr2=0x7f1ddc1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1ddaffd640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc102250 0x7f1ddc1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1ddaffd640 1 -- 172.21.15.64:0/1483200572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ddc19d130 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1dda7fc640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc102250 0x7f1ddc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1ddaffd640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ddc1018b0 0x7f1ddc198180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc005940 tx=0x7f1dcc031b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1dc3fff640 1 -- 172.21.15.64:0/1483200572 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc03f070 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ddc19d3c0 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1dc3fff640 1 -- 172.21.15.64:0/1483200572 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dcc003ad0 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1dc3fff640 1 -- 172.21.15.64:0/1483200572 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc035070 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.182+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ddc1a4d00 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.183+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ddc101d10 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.185+0000 7f1dc3fff640 1 -- 172.21.15.64:0/1483200572 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1dcc0035f0 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.185+0000 7f1dc3fff640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1dc40a2d70 0x7f1dc40a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.186+0000 7f1dda7fc640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1dc40a2d70 0x7f1dc40a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.186+0000 7f1dc3fff640 1 -- 172.21.15.64:0/1483200572 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f1dcc0efe80 con 0x7f1ddc1018b0 2024-08-19T01:41:01.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.186+0000 7f1dda7fc640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1dc40a2d70 0x7f1dc40a5230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0fffe0 tx=0x7f1dd0005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:01.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.189+0000 7f1dc3fff640 1 -- 172.21.15.64:0/1483200572 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1dcc0bc040 con 0x7f1ddc1018b0 2024-08-19T01:41:01.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 723771 9.34 MB/sec execute 273 sec latency 87.588 ms 2024-08-19T01:41:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.418+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ddc1994e0 con 0x7f1ddc1018b0 2024-08-19T01:41:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.419+0000 7f1dc3fff640 1 -- 172.21.15.64:0/1483200572 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f1dcc0c0340 con 0x7f1ddc1018b0 2024-08-19T01:41:01.422 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:41:01.422 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:41:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:41:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1dc40a2d70 msgr2=0x7f1dc40a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1dc40a2d70 0x7f1dc40a5230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0fffe0 tx=0x7f1dd0005f90 comp rx=0 tx=0).stop 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ddc1018b0 msgr2=0x7f1ddc198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ddc1018b0 0x7f1ddc198180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc005940 tx=0x7f1dcc031b50 comp rx=0 tx=0).stop 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 shutdown_connections 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1dc40a2d70 0x7f1dc40a5230 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ddc1078b0 0x7f1ddc19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc102250 0x7f1ddc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 --2- 172.21.15.64:0/1483200572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ddc1018b0 0x7f1ddc198180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 >> 172.21.15.64:0/1483200572 conn(0x7f1ddc0fd5b0 msgr2=0x7f1ddc0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.421+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 shutdown_connections 2024-08-19T01:41:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:01.422+0000 7f1de17a9640 1 -- 172.21.15.64:0/1483200572 wait complete. 2024-08-19T01:41:02.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:01 smithi123 ceph-mon[21829]: pgmap v238: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.5 MiB/s wr, 320 op/s 2024-08-19T01:41:02.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:01 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1483200572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:02.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:01 smithi161 ceph-mon[21757]: pgmap v238: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.5 MiB/s wr, 320 op/s 2024-08-19T01:41:02.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:01 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1483200572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:02.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:01 smithi064 ceph-mon[18515]: pgmap v238: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.5 MiB/s wr, 320 op/s 2024-08-19T01:41:02.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:01 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1483200572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:02.319 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 143.75 --await-for 45.0' 2024-08-19T01:41:02.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 143.75 --await-for 45.0 2024-08-19T01:41:02.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 725651 9.34 MB/sec execute 274 sec latency 57.995 ms 2024-08-19T01:41:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:03.099 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:02 smithi161 ceph-mon[21757]: pgmap v239: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-08-19T01:41:03.159 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:02 smithi064 ceph-mon[18515]: pgmap v239: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-08-19T01:41:03.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:02 smithi123 ceph-mon[21829]: pgmap v239: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 -- 172.21.15.64:0/481000237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d8100b50 msgr2=0x7ff4d8100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 --2- 172.21.15.64:0/481000237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d8100b50 0x7ff4d8100f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009900 tx=0x7ff4c802a480 comp rx=0 tx=0).stop 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 -- 172.21.15.64:0/481000237 shutdown_connections 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 --2- 172.21.15.64:0/481000237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8102c20 0x7ff4d810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 --2- 172.21.15.64:0/481000237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4d81022a0 0x7ff4d81026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 --2- 172.21.15.64:0/481000237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d8100b50 0x7ff4d8100f30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 -- 172.21.15.64:0/481000237 >> 172.21.15.64:0/481000237 conn(0x7ff4d80fc500 msgr2=0x7ff4d80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 -- 172.21.15.64:0/481000237 shutdown_connections 2024-08-19T01:41:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.252+0000 7ff4de88e640 1 -- 172.21.15.64:0/481000237 wait complete. 2024-08-19T01:41:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 Processor -- start 2024-08-19T01:41:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 -- start start 2024-08-19T01:41:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8100b50 0x7ff4d8103480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d81022a0 0x7ff4d81039c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4d8102c20 0x7ff4d8103f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d8110cb0 con 0x7ff4d81022a0 2024-08-19T01:41:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d8110b30 con 0x7ff4d8100b50 2024-08-19T01:41:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4de88e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d8110e30 con 0x7ff4d8102c20 2024-08-19T01:41:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4d7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8100b50 0x7ff4d8103480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4d7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8100b50 0x7ff4d8103480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:42142/0 (socket says 172.21.15.64:42142) 2024-08-19T01:41:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.253+0000 7ff4d7fff640 1 -- 172.21.15.64:0/1021768065 learned_addr learned my addr 172.21.15.64:0/1021768065 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4d8102c20 0x7ff4d8103f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d81022a0 0x7ff4d81039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d7fff640 1 -- 172.21.15.64:0/1021768065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4d8102c20 msgr2=0x7ff4d8103f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d7fff640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4d8102c20 0x7ff4d8103f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d7fff640 1 -- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d81022a0 msgr2=0x7ff4d81039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d7fff640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d81022a0 0x7ff4d81039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d7fff640 1 -- 172.21.15.64:0/1021768065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d81a6710 con 0x7ff4d8100b50 2024-08-19T01:41:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4d8102c20 0x7ff4d8103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d81022a0 0x7ff4d81039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d7fff640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8100b50 0x7ff4d8103480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8006010 tx=0x7ff4c8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d57fa640 1 -- 172.21.15.64:0/1021768065 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c803f070 con 0x7ff4d8100b50 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d57fa640 1 -- 172.21.15.64:0/1021768065 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c8031c30 con 0x7ff4d8100b50 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d81a69a0 con 0x7ff4d8100b50 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4d57fa640 1 -- 172.21.15.64:0/1021768065 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8035030 con 0x7ff4d8100b50 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.254+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d81a6e80 con 0x7ff4d8100b50 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4d57fa640 1 -- 172.21.15.64:0/1021768065 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff4c8042070 con 0x7ff4d8100b50 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 Processor -- start 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 -- start start 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166372c0 0x556f166376a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f16637be0 0x556f16641f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16644890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556f1659c810 con 0x556f16637be0 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x556f16543c70 con 0x556f166372c0 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556f165987d0 con 0x556f166424a0 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4d7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f16637be0 0x556f16641f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166372c0 0x556f166376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166372c0 0x556f166376a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:42158/0 (socket says 172.21.15.64:42158) 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2745731575 learned_addr learned my addr 172.21.15.64:0/2745731575 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.257+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16644890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2745731575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 msgr2=0x556f16644890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16644890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2745731575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f16637be0 msgr2=0x556f16641f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f16637be0 0x556f16641f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2745731575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1630eba0 con 0x556f166372c0 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4d57fa640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff4c00a2cf0 0x7ff4c00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166372c0 0x556f166376a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0010710 tx=0x7ff4d0010bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef2665df473e988d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4d7fff640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff4c00a2cf0 0x7ff4c00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4d57fa640 1 -- 172.21.15.64:0/1021768065 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7ff4c80efeb0 con 0x7ff4d8100b50 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4be7fc640 1 -- 172.21.15.64:0/2745731575 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d000e8d0 con 0x556f166372c0 2024-08-19T01:41:03.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4be7fc640 1 -- 172.21.15.64:0/2745731575 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0012450 con 0x556f166372c0 2024-08-19T01:41:03.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.258+0000 7ff4be7fc640 1 -- 172.21.15.64:0/2745731575 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d001c650 con 0x556f166372c0 2024-08-19T01:41:03.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2745731575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166372c0 msgr2=0x556f166376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4d7fff640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff4c00a2cf0 0x7ff4c00a51b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80ed2a0 tx=0x7ff4c80f3a00 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:03.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166372c0 0x556f166376a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0010710 tx=0x7ff4d0010bd0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2745731575 shutdown_connections 2024-08-19T01:41:03.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16644890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f16637be0 0x556f16641f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2745731575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166372c0 0x556f166376a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2745731575 >> 172.21.15.64:0/2745731575 conn(0x556f16633850 msgr2=0x556f16633c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2745731575 shutdown_connections 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.259+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2745731575 wait complete. 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 -- start start 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 Processor -- start 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1663f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f1663fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16640010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556f16644fb0 con 0x556f166372c0 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x556f165579a0 con 0x556f16637be0 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556f16645130 con 0x556f166424a0 2024-08-19T01:41:03.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4e0c8c740 10 client.0 _ll_register_callbacks cb 0x556f16545c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1663f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1663f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33176/0 (socket says 172.21.15.64:33176) 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2206914629 learned_addr learned my addr 172.21.15.64:0/2206914629 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f1663fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.260+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16640010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2206914629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 msgr2=0x556f16640010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16640010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 msgr2=0x556f1663fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f1663fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4dce2b640 1 -- 172.21.15.64:0/2206914629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f16563220 con 0x556f166372c0 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1663f590 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0012ed0 tx=0x7ff4d0009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1605dd275e14e04 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4b77fe640 1 -- 172.21.15.64:0/2206914629 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d000e8d0 con 0x556f166372c0 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4b77fe640 1 -- 172.21.15.64:0/2206914629 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7ff4d000ea70 con 0x556f166372c0 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d0012450 con 0x556f166372c0 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4b77fe640 1 -- 172.21.15.64:0/2206914629 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff4d000ec70 con 0x556f166372c0 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4e0c8c740 10 client.15777 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:03.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f16650de0 con 0x556f166372c0 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.261+0000 7ff4b77fe640 1 -- 172.21.15.64:0/2206914629 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0012450 con 0x556f166372c0 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4b77fe640 1 -- 172.21.15.64:0/2206914629 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ff4d0029070 con 0x556f166372c0 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4e0c8c740 10 client.15777 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4e0c8c740 10 client.15777 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4e0c8c740 20 client.15777 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166521f0 0x556f166545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4e0c8c740 4 client.15777 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:41:03.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f165a9f40 con 0x556f166521f0 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166521f0 0x556f166545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166521f0 0x556f166545e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=521c57fc3d38ff50 server_cookie=97f844d9f901dc71 in_seq=0 out_seq=0 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.262+0000 7ff4b77fe640 10 client.15777 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4b77fe640 1 -- 172.21.15.64:0/2206914629 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556f165a9f40 con 0x556f166521f0 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4b77fe640 10 client.15777 handle_command_reply: tid=0 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 client.15777 shutdown 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 msgr2=0x556f1663f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1663f590 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0012ed0 tx=0x7ff4d0009d10 comp rx=0 tx=0).stop 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 shutdown_connections 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16640010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166521f0 0x556f166545e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f1663fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/2206914629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1663f590 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4b77fe640 0 client.15777 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:03.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 >> 172.21.15.64:0/2206914629 conn(0x556f16633850 msgr2=0x556f16632b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:03.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 shutdown_connections 2024-08-19T01:41:03.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/2206914629 wait complete. 2024-08-19T01:41:03.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.263+0000 7ff4e0c8c740 20 client.15777 trim_cache size 0 max 16384 2024-08-19T01:41:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 Processor -- start 2024-08-19T01:41:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 -- start start 2024-08-19T01:41:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f16632a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f16637be0 0x556f16659d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166424a0 0x556f1665a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81b39f0 con 0x556f166372c0 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81b12c0 con 0x556f166424a0 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81af130 con 0x556f16637be0 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f16632a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f16632a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33190/0 (socket says 172.21.15.64:33190) 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1385880302 learned_addr learned my addr 172.21.15.64:0/1385880302 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f16637be0 0x556f16659d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166424a0 0x556f1665a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1385880302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f16637be0 msgr2=0x556f16659d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f16637be0 0x556f16659d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1385880302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166424a0 msgr2=0x556f1665a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166424a0 0x556f1665a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.281+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1385880302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1664e430 con 0x556f166372c0 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.282+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f16632a20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00104d0 tx=0x7ff4d0010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1a8f1654e9b7d0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.282+0000 7ff4bdffb640 1 -- 172.21.15.64:0/1385880302 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0040070 con 0x556f166372c0 2024-08-19T01:41:03.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.282+0000 7ff4bdffb640 1 -- 172.21.15.64:0/1385880302 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0026040 con 0x556f166372c0 2024-08-19T01:41:03.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.282+0000 7ff4bdffb640 1 -- 172.21.15.64:0/1385880302 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d003ba80 con 0x556f166372c0 2024-08-19T01:41:03.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1385880302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 msgr2=0x556f16632a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f16632a20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00104d0 tx=0x7ff4d0010bd0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1385880302 shutdown_connections 2024-08-19T01:41:03.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f166424a0 0x556f1665a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f16637be0 0x556f16659d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1385880302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f16632a20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1385880302 >> 172.21.15.64:0/1385880302 conn(0x556f16633850 msgr2=0x556f16632310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1385880302 shutdown_connections 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1385880302 wait complete. 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 -- start start 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.283+0000 7ff4e0c8c740 1 Processor -- start 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1664a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f16645a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4e0c8c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16645fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81af780 con 0x556f166372c0 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81b0bf0 con 0x556f16637be0 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4e0c8c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81afe50 con 0x556f166424a0 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4e0c8c740 10 client.0 _ll_register_callbacks cb 0x7ff4d0016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:03.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1664a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4d77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f16645a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1664a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33200/0 (socket says 172.21.15.64:33200) 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1728297869 learned_addr learned my addr 172.21.15.64:0/1728297869 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16645fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1728297869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 msgr2=0x556f16645fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16645fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1728297869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 msgr2=0x556f16645a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.284+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f16645a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4dce2b640 1 -- 172.21.15.64:0/1728297869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f166464e0 con 0x556f166372c0 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4dce2b640 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1664a690 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00129b0 tx=0x7ff4d003b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91181b1446d5b1cd server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:03.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4b77fe640 1 -- 172.21.15.64:0/1728297869 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0040040 con 0x556f166372c0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4b77fe640 1 -- 172.21.15.64:0/1728297869 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7ff4d003ba00 con 0x556f166372c0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d0026040 con 0x556f166372c0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4b77fe640 1 -- 172.21.15.64:0/1728297869 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff4d004f050 con 0x556f166372c0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4e0c8c740 10 client.15783 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f166c5190 con 0x556f166372c0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.285+0000 7ff4b77fe640 1 -- 172.21.15.64:0/1728297869 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d003bc00 con 0x556f166372c0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4b77fe640 1 -- 172.21.15.64:0/1728297869 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ff4d0046040 con 0x556f166372c0 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4e0c8c740 10 client.15783 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4e0c8c740 10 client.15783 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4e0c8c740 20 client.15783 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166c5680 0x556f166c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:03.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4e0c8c740 4 client.15783 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.75, "await_for": 45.0}] 2024-08-19T01:41:03.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.75, "await_for": 45.0}) -- 0x7ff4d0026040 con 0x556f166c5680 2024-08-19T01:41:03.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166c5680 0x556f166c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:03.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4b77fe640 10 client.15783 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:03.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.286+0000 7ff4d77fe640 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166c5680 0x556f166c7a70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=425fb188a78375a3 server_cookie=5ce1f78f7b0163da in_seq=0 out_seq=0 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4b77fe640 1 -- 172.21.15.64:0/1728297869 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7ff4d0026040 con 0x556f166c5680 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4b77fe640 10 client.15783 handle_command_reply: tid=0 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 client.15783 shutdown 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 msgr2=0x556f1664a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1664a690 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00129b0 tx=0x7ff4d003b660 comp rx=0 tx=0).stop 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 shutdown_connections 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556f166424a0 0x556f16645fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x556f166c5680 0x556f166c7a70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x556f16637be0 0x556f16645a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 --2- 172.21.15.64:0/1728297869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556f166372c0 0x556f1664a690 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4b77fe640 0 client.15783 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 >> 172.21.15.64:0/1728297869 conn(0x556f16633850 msgr2=0x556f16632b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.366+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 shutdown_connections 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4e0c8c740 1 -- 172.21.15.64:0/1728297869 wait complete. 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4e0c8c740 20 client.15783 trim_cache size 0 max 16384 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 26, 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:41:03.370 INFO:teuthology.orchestra.run.smithi064.stdout: "fcdc5a6a": { 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 26, 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 143.8, 2024-08-19T01:41:03.371 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:03.372 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:41:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff4c00a2cf0 msgr2=0x7ff4c00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff4c00a2cf0 0x7ff4c00a51b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80ed2a0 tx=0x7ff4c80f3a00 comp rx=0 tx=0).stop 2024-08-19T01:41:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8100b50 msgr2=0x7ff4d8103480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8100b50 0x7ff4d8103480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8006010 tx=0x7ff4c8003c10 comp rx=0 tx=0).stop 2024-08-19T01:41:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 shutdown_connections 2024-08-19T01:41:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff4c00a2cf0 0x7ff4c00a51b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4d8102c20 0x7ff4d8103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4d81022a0 0x7ff4d81039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 --2- 172.21.15.64:0/1021768065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d8100b50 0x7ff4d8103480 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 >> 172.21.15.64:0/1021768065 conn(0x7ff4d80fc500 msgr2=0x7ff4d8069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 shutdown_connections 2024-08-19T01:41:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:03.367+0000 7ff4de88e640 1 -- 172.21.15.64:0/1021768065 wait complete. 2024-08-19T01:41:03.395 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.34 MB/sec execute 275 sec latency 162.482 ms 2024-08-19T01:41:04.163 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'fcdc5a6a', quiesce took 3.9 seconds. Will release after: 19.85 2024-08-19T01:41:04.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.31 MB/sec execute 276 sec latency 1085.061 ms 2024-08-19T01:41:05.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.27 MB/sec execute 277 sec latency 2085.160 ms 2024-08-19T01:41:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:04 smithi123 ceph-mon[21829]: pgmap v240: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-08-19T01:41:05.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:04 smithi161 ceph-mon[21757]: pgmap v240: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-08-19T01:41:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:04 smithi064 ceph-mon[18515]: pgmap v240: 129 pgs: 129 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-08-19T01:41:06.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.24 MB/sec execute 278 sec latency 3085.244 ms 2024-08-19T01:41:07.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.21 MB/sec execute 279 sec latency 4085.339 ms 2024-08-19T01:41:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:06 smithi123 ceph-mon[21829]: pgmap v241: 129 pgs: 129 active+clean; 473 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 319 op/s 2024-08-19T01:41:07.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:07 smithi161 ceph-mon[21757]: pgmap v241: 129 pgs: 129 active+clean; 473 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 319 op/s 2024-08-19T01:41:07.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:06 smithi064 ceph-mon[18515]: pgmap v241: 129 pgs: 129 active+clean; 473 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 319 op/s 2024-08-19T01:41:08.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.17 MB/sec execute 280 sec latency 5085.455 ms 2024-08-19T01:41:08.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:08 smithi123 ceph-mon[21829]: osdmap e97: 12 total, 12 up, 12 in 2024-08-19T01:41:08.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:08 smithi161 ceph-mon[21757]: osdmap e97: 12 total, 12 up, 12 in 2024-08-19T01:41:08.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:08 smithi064 ceph-mon[18515]: osdmap e97: 12 total, 12 up, 12 in 2024-08-19T01:41:09.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.14 MB/sec execute 281 sec latency 6085.540 ms 2024-08-19T01:41:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:09 smithi123 ceph-mon[21829]: pgmap v243: 129 pgs: 129 active+clean; 473 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 261 op/s 2024-08-19T01:41:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:09 smithi161 ceph-mon[21757]: pgmap v243: 129 pgs: 129 active+clean; 473 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 261 op/s 2024-08-19T01:41:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:09 smithi064 ceph-mon[18515]: pgmap v243: 129 pgs: 129 active+clean; 473 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 261 op/s 2024-08-19T01:41:10.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.11 MB/sec execute 282 sec latency 7085.590 ms 2024-08-19T01:41:11.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.08 MB/sec execute 283 sec latency 8085.697 ms 2024-08-19T01:41:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:11 smithi123 ceph-mon[21829]: pgmap v244: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.7 MiB/s wr, 146 op/s 2024-08-19T01:41:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:11 smithi161 ceph-mon[21757]: pgmap v244: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.7 MiB/s wr, 146 op/s 2024-08-19T01:41:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:11 smithi064 ceph-mon[18515]: pgmap v244: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.7 MiB/s wr, 146 op/s 2024-08-19T01:41:12.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.04 MB/sec execute 284 sec latency 9085.808 ms 2024-08-19T01:41:13.396 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 9.01 MB/sec execute 285 sec latency 10085.898 ms 2024-08-19T01:41:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:13 smithi123 ceph-mon[21829]: pgmap v245: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.7 MiB/s wr, 146 op/s 2024-08-19T01:41:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:13 smithi161 ceph-mon[21757]: pgmap v245: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.7 MiB/s wr, 146 op/s 2024-08-19T01:41:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:13 smithi064 ceph-mon[18515]: pgmap v245: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.7 MiB/s wr, 146 op/s 2024-08-19T01:41:14.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.98 MB/sec execute 286 sec latency 11085.988 ms 2024-08-19T01:41:14.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:14 smithi064 ceph-mon[18515]: osdmap e98: 12 total, 12 up, 12 in 2024-08-19T01:41:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:14 smithi123 ceph-mon[21829]: osdmap e98: 12 total, 12 up, 12 in 2024-08-19T01:41:15.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:14 smithi161 ceph-mon[21757]: osdmap e98: 12 total, 12 up, 12 in 2024-08-19T01:41:15.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.95 MB/sec execute 287 sec latency 12086.055 ms 2024-08-19T01:41:15.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:15 smithi064 ceph-mon[18515]: pgmap v247: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 MiB/s wr, 24 op/s 2024-08-19T01:41:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:15 smithi123 ceph-mon[21829]: pgmap v247: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 MiB/s wr, 24 op/s 2024-08-19T01:41:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:15 smithi161 ceph-mon[21757]: pgmap v247: 129 pgs: 129 active+clean; 478 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 MiB/s wr, 24 op/s 2024-08-19T01:41:16.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.92 MB/sec execute 288 sec latency 13086.155 ms 2024-08-19T01:41:17.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.89 MB/sec execute 289 sec latency 14086.265 ms 2024-08-19T01:41:17.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:17 smithi064 ceph-mon[18515]: pgmap v248: 129 pgs: 129 active+clean; 477 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.0 MiB/s wr, 64 op/s 2024-08-19T01:41:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:17 smithi123 ceph-mon[21829]: pgmap v248: 129 pgs: 129 active+clean; 477 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.0 MiB/s wr, 64 op/s 2024-08-19T01:41:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:17 smithi161 ceph-mon[21757]: pgmap v248: 129 pgs: 129 active+clean; 477 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 2.0 MiB/s wr, 64 op/s 2024-08-19T01:41:18.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.86 MB/sec execute 290 sec latency 15086.326 ms 2024-08-19T01:41:19.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.83 MB/sec execute 291 sec latency 16086.409 ms 2024-08-19T01:41:19.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:19 smithi064 ceph-mon[18515]: pgmap v249: 129 pgs: 129 active+clean; 477 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.8 MiB/s wr, 57 op/s 2024-08-19T01:41:20.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:19 smithi123 ceph-mon[21829]: pgmap v249: 129 pgs: 129 active+clean; 477 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.8 MiB/s wr, 57 op/s 2024-08-19T01:41:20.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:19 smithi161 ceph-mon[21757]: pgmap v249: 129 pgs: 129 active+clean; 477 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 1.8 MiB/s wr, 57 op/s 2024-08-19T01:41:20.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.80 MB/sec execute 292 sec latency 17086.509 ms 2024-08-19T01:41:21.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.77 MB/sec execute 293 sec latency 18086.614 ms 2024-08-19T01:41:21.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:21 smithi064 ceph-mon[18515]: pgmap v250: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-19T01:41:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:21 smithi123 ceph-mon[21829]: pgmap v250: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-19T01:41:22.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:21 smithi161 ceph-mon[21757]: pgmap v250: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-19T01:41:22.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.74 MB/sec execute 294 sec latency 19086.685 ms 2024-08-19T01:41:22.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:22 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:41:23.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:22 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:41:23.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:22 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:41:23.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.71 MB/sec execute 295 sec latency 20086.764 ms 2024-08-19T01:41:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:23 smithi064 ceph-mon[18515]: pgmap v251: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-19T01:41:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:23 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:41:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:23 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:41:23.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:23 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:41:24.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:41:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:23 smithi123 ceph-mon[21829]: pgmap v251: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-19T01:41:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:23 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:41:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:23 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:41:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:23 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:41:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:23 smithi161 ceph-mon[21757]: pgmap v251: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-08-19T01:41:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:23 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:41:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:23 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:41:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:23 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:41:24.282 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:24.397 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.68 MB/sec execute 296 sec latency 21086.872 ms 2024-08-19T01:41:24.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 -- 172.21.15.64:0/2425645227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 msgr2=0x7eff28101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:24.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/2425645227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28101d10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7eff18009930 tx=0x7eff1802a420 comp rx=0 tx=0).stop 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 -- 172.21.15.64:0/2425645227 shutdown_connections 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/2425645227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff28102250 0x7eff2810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/2425645227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28101d10 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/2425645227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff281078b0 0x7eff28107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 -- 172.21.15.64:0/2425645227 >> 172.21.15.64:0/2425645227 conn(0x7eff280fd5f0 msgr2=0x7eff280ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.750+0000 7eff2f7eb640 1 -- 172.21.15.64:0/2425645227 shutdown_connections 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 -- 172.21.15.64:0/2425645227 wait complete. 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 Processor -- start 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 -- start start 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff28102250 0x7eff28198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:24.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff281078b0 0x7eff2819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7eff28111c70 con 0x7eff281018b0 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7eff28111af0 con 0x7eff281078b0 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2f7eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff28111df0 con 0x7eff28102250 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2d587640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2d587640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41714/0 (socket says 172.21.15.64:41714) 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.751+0000 7eff2d587640 1 -- 172.21.15.64:0/3650088708 learned_addr learned my addr 172.21.15.64:0/3650088708 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2cd86640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff28102250 0x7eff28198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2dd88640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff281078b0 0x7eff2819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2d587640 1 -- 172.21.15.64:0/3650088708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff28102250 msgr2=0x7eff28198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2d587640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff28102250 0x7eff28198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2d587640 1 -- 172.21.15.64:0/3650088708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff281078b0 msgr2=0x7eff2819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2d587640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff281078b0 0x7eff2819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2d587640 1 -- 172.21.15.64:0/3650088708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2819d0e0 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2cd86640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff28102250 0x7eff28198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2d587640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28198130 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c920 tx=0x7eff2400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff1e7fc640 1 -- 172.21.15.64:0/3650088708 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24007c70 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff1e7fc640 1 -- 172.21.15.64:0/3650088708 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff24007e10 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff1e7fc640 1 -- 172.21.15.64:0/3650088708 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24018890 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff2819d3d0 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.752+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff281a4cb0 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.753+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff28101d10 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.754+0000 7eff1e7fc640 1 -- 172.21.15.64:0/3650088708 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7eff24018a70 con 0x7eff281018b0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.755+0000 7eff1e7fc640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7eff140a2d70 0x7eff140a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.755+0000 7eff2cd86640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7eff140a2d70 0x7eff140a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:24.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.755+0000 7eff1e7fc640 1 -- 172.21.15.64:0/3650088708 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7eff240cf950 con 0x7eff281018b0 2024-08-19T01:41:24.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.755+0000 7eff2cd86640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7eff140a2d70 0x7eff140a5230 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7eff18000c00 tx=0x7eff18005a20 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:24.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.759+0000 7eff1e7fc640 1 -- 172.21.15.64:0/3650088708 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7eff24014030 con 0x7eff281018b0 2024-08-19T01:41:24.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.971+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff28199490 con 0x7eff281018b0 2024-08-19T01:41:24.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.972+0000 7eff1e7fc640 1 -- 172.21.15.64:0/3650088708 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7eff2401f300 con 0x7eff281018b0 2024-08-19T01:41:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:41:24.974 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:41:24.974 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:41:24.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7eff140a2d70 msgr2=0x7eff140a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:24.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7eff140a2d70 0x7eff140a5230 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7eff18000c00 tx=0x7eff18005a20 comp rx=0 tx=0).stop 2024-08-19T01:41:24.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 msgr2=0x7eff28198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:24.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28198130 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c920 tx=0x7eff2400cde0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 shutdown_connections 2024-08-19T01:41:24.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7eff140a2d70 0x7eff140a5230 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff281078b0 0x7eff2819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff28102250 0x7eff28198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 --2- 172.21.15.64:0/3650088708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff281018b0 0x7eff28198130 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:24.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 >> 172.21.15.64:0/3650088708 conn(0x7eff280fd5f0 msgr2=0x7eff280fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:24.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 shutdown_connections 2024-08-19T01:41:24.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:24.974+0000 7eff2f7eb640 1 -- 172.21.15.64:0/3650088708 wait complete. 2024-08-19T01:41:25.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.65 MB/sec execute 297 sec latency 22086.972 ms 2024-08-19T01:41:25.719 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id fcdc5a6a --release --await' 2024-08-19T01:41:25.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id fcdc5a6a --release --await 2024-08-19T01:41:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:25.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:25 smithi064 ceph-mon[18515]: pgmap v252: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-08-19T01:41:25.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:25 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3650088708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:25 smithi123 ceph-mon[21829]: pgmap v252: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-08-19T01:41:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:25 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3650088708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:25 smithi161 ceph-mon[21757]: pgmap v252: 129 pgs: 129 active+clean; 439 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-08-19T01:41:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:25 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3650088708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:26.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 727283 8.62 MB/sec execute 298 sec latency 23087.013 ms 2024-08-19T01:41:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 -- 172.21.15.64:0/3731994182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 msgr2=0x7f96ec107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 --2- 172.21.15.64:0/3731994182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec107c80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009c10 tx=0x7f96e402a6e0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 -- 172.21.15.64:0/3731994182 shutdown_connections 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 --2- 172.21.15.64:0/3731994182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96ec102240 0x7f96ec10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 --2- 172.21.15.64:0/3731994182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96ec1018a0 0x7f96ec101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 --2- 172.21.15.64:0/3731994182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec107c80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 -- 172.21.15.64:0/3731994182 >> 172.21.15.64:0/3731994182 conn(0x7f96ec0fd5e0 msgr2=0x7f96ec0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 -- 172.21.15.64:0/3731994182 shutdown_connections 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.425+0000 7f96f1795640 1 -- 172.21.15.64:0/3731994182 wait complete. 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 Processor -- start 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 -- start start 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96ec1018a0 0x7f96ec198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96ec102240 0x7f96ec1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec111c50 con 0x7f96ec1078a0 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec111ad0 con 0x7f96ec1018a0 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96f1795640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec111dd0 con 0x7f96ec102240 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41728/0 (socket says 172.21.15.64:41728) 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96eb7fe640 1 -- 172.21.15.64:0/494810983 learned_addr learned my addr 172.21.15.64:0/494810983 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96eaffd640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96ec1018a0 0x7f96ec198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.426+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96ec102240 0x7f96ec1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96eb7fe640 1 -- 172.21.15.64:0/494810983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96ec102240 msgr2=0x7f96ec1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96eb7fe640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96ec102240 0x7f96ec1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96eb7fe640 1 -- 172.21.15.64:0/494810983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96ec1018a0 msgr2=0x7f96ec198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96eb7fe640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96ec1018a0 0x7f96ec198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96eb7fe640 1 -- 172.21.15.64:0/494810983 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ec19d120 con 0x7f96ec1078a0 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96eb7fe640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec19ca40 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00e970 tx=0x7f96dc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96e3fff640 1 -- 172.21.15.64:0/494810983 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc00ccc0 con 0x7f96ec1078a0 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96e3fff640 1 -- 172.21.15.64:0/494810983 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96dc005500 con 0x7f96ec1078a0 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ec19d410 con 0x7f96ec1078a0 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96e3fff640 1 -- 172.21.15.64:0/494810983 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc010690 con 0x7f96ec1078a0 2024-08-19T01:41:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.427+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ec1a4cf0 con 0x7f96ec1078a0 2024-08-19T01:41:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.429+0000 7f96e3fff640 1 -- 172.21.15.64:0/494810983 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f96dc004e50 con 0x7f96ec1078a0 2024-08-19T01:41:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 Processor -- start 2024-08-19T01:41:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 -- start start 2024-08-19T01:41:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39dda0 0x55cefa39e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa39e6c0 0x55cefa3a8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa3a8f80 0x55cefa3ab370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cefa2ffc60 con 0x55cefa39e6c0 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cefa297b90 con 0x55cefa3a8f80 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96f3b93740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cefa2fbc20 con 0x55cefa39dda0 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96eaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa39e6c0 0x55cefa3a8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96eaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa39e6c0 0x55cefa3a8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41740/0 (socket says 172.21.15.64:41740) 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96eaffd640 1 -- 172.21.15.64:0/983514955 learned_addr learned my addr 172.21.15.64:0/983514955 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.430+0000 7f96e3fff640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f96d80a2cf0 0x7f96d80a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39dda0 0x55cefa39e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa3a8f80 0x55cefa3ab370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eaffd640 1 -- 172.21.15.64:0/983514955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39dda0 msgr2=0x55cefa39e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eaffd640 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39dda0 0x55cefa39e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eb7fe640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f96d80a2cf0 0x7f96d80a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eaffd640 1 -- 172.21.15.64:0/983514955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa3a8f80 msgr2=0x55cefa3ab370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eaffd640 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa3a8f80 0x55cefa3ab370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eaffd640 1 -- 172.21.15.64:0/983514955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cefa00f580 con 0x55cefa39e6c0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39dda0 0x55cefa39e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96e3fff640 1 -- 172.21.15.64:0/494810983 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f96dc0cfaf0 con 0x7f96ec1078a0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eaffd640 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa39e6c0 0x55cefa3a8a40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009310 tx=0x7f96e4005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30aef7839b3c8bbf server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96eb7fe640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f96d80a2cf0 0x7f96d80a51b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c6a0 tx=0x7f96dc097840 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96e17fa640 1 -- 172.21.15.64:0/983514955 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4042030 con 0x55cefa39e6c0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96e17fa640 1 -- 172.21.15.64:0/983514955 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e4037040 con 0x55cefa39e6c0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.431+0000 7f96e17fa640 1 -- 172.21.15.64:0/983514955 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4036070 con 0x55cefa39e6c0 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 -- 172.21.15.64:0/983514955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa39e6c0 msgr2=0x55cefa3a8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa39e6c0 0x55cefa3a8a40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009310 tx=0x7f96e4005c60 comp rx=0 tx=0).stop 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 -- 172.21.15.64:0/983514955 shutdown_connections 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa3a8f80 0x55cefa3ab370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa39e6c0 0x55cefa3a8a40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 --2- 172.21.15.64:0/983514955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39dda0 0x55cefa39e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 -- 172.21.15.64:0/983514955 >> 172.21.15.64:0/983514955 conn(0x55cefa39a330 msgr2=0x55cefa39a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 -- 172.21.15.64:0/983514955 shutdown_connections 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.432+0000 7f96f3b93740 1 -- 172.21.15.64:0/983514955 wait complete. 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 -- start start 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 Processor -- start 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3a6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55cefa3abb50 con 0x55cefa3a8f80 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cefa29d7c0 con 0x55cefa39dda0 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cefa3abcd0 con 0x55cefa39e6c0 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96f3b93740 10 client.0 _ll_register_callbacks cb 0x55cefa2a6f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41752/0 (socket says 172.21.15.64:41752) 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 -- 172.21.15.64:0/2458332052 learned_addr learned my addr 172.21.15.64:0/2458332052 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96eaffd640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 -- 172.21.15.64:0/2458332052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 msgr2=0x55cefa3a6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3a6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 -- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 msgr2=0x55cefa3a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96ea7fc640 1 -- 172.21.15.64:0/2458332052 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cefa0d9960 con 0x55cefa3a8f80 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.433+0000 7f96eaffd640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3a68b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f96d4010ec0 tx=0x7f96d4009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92df59ecbfccc39 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96d27fc640 1 -- 172.21.15.64:0/2458332052 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4019040 con 0x55cefa3a8f80 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96d27fc640 1 -- 172.21.15.64:0/2458332052 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f96d4005220 con 0x55cefa3a8f80 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e4037040 con 0x55cefa3a8f80 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96d27fc640 1 -- 172.21.15.64:0/2458332052 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f96d4029050 con 0x55cefa3a8f80 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 10 client.15801 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cefa3b7b60 con 0x55cefa3a8f80 2024-08-19T01:41:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96d27fc640 1 -- 172.21.15.64:0/2458332052 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e4037040 con 0x55cefa3a8f80 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96d27fc640 1 -- 172.21.15.64:0/2458332052 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f96d401f070 con 0x55cefa3a8f80 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 10 client.15801 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 10 client.15801 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 20 client.15801 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa3b8f70 0x55cefa3bb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 4 client.15801 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.434+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cefa30d1d0 con 0x55cefa3b8f70 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.435+0000 7f96eaffd640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa3b8f70 0x55cefa3bb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.435+0000 7f96eaffd640 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa3b8f70 0x55cefa3bb3a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d71d6ad0bc2d408f server_cookie=2d0e59915dfc0a13 in_seq=0 out_seq=0 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.435+0000 7f96d27fc640 10 client.15801 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96d27fc640 1 -- 172.21.15.64:0/2458332052 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cefa30d1d0 con 0x55cefa3b8f70 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96d27fc640 10 client.15801 handle_command_reply: tid=0 2024-08-19T01:41:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 client.15801 shutdown 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 msgr2=0x55cefa3a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3a68b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f96d4010ec0 tx=0x7f96d4009b40 comp rx=0 tx=0).stop 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 shutdown_connections 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3a68b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa3b8f70 0x55cefa3bb3a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 --2- 172.21.15.64:0/2458332052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96d27fc640 0 client.15801 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 >> 172.21.15.64:0/2458332052 conn(0x55cefa39a330 msgr2=0x55cefa395080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 shutdown_connections 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 1 -- 172.21.15.64:0/2458332052 wait complete. 2024-08-19T01:41:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.436+0000 7f96f3b93740 20 client.15801 trim_cache size 0 max 16384 2024-08-19T01:41:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.453+0000 7f96f3b93740 1 Processor -- start 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.453+0000 7f96f3b93740 1 -- start start 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3c0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96f3b93740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1af510 con 0x55cefa3a8f80 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96f3b93740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1acde0 con 0x55cefa39dda0 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96f3b93740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1aac50 con 0x55cefa39e6c0 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3c11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41762/0 (socket says 172.21.15.64:41762) 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 -- 172.21.15.64:0/4036748174 learned_addr learned my addr 172.21.15.64:0/4036748174 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96eaffd640 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3c0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 -- 172.21.15.64:0/4036748174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 msgr2=0x55cefa3c0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3c0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 -- 172.21.15.64:0/4036748174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 msgr2=0x55cefa3a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 -- 172.21.15.64:0/4036748174 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cefa3a4de0 con 0x55cefa3a8f80 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3c11b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f96d4001eb0 tx=0x7f96d400ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07b3f3214faa343 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96e0ff9640 1 -- 172.21.15.64:0/4036748174 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d400e980 con 0x55cefa3a8f80 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96e0ff9640 1 -- 172.21.15.64:0/4036748174 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d4005d30 con 0x55cefa3a8f80 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.454+0000 7f96e0ff9640 1 -- 172.21.15.64:0/4036748174 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d40049b0 con 0x55cefa3a8f80 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 -- 172.21.15.64:0/4036748174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 msgr2=0x55cefa3c11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3c11b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f96d4001eb0 tx=0x7f96d400ff90 comp rx=0 tx=0).stop 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 -- 172.21.15.64:0/4036748174 shutdown_connections 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3c11b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3c0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 --2- 172.21.15.64:0/4036748174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 -- 172.21.15.64:0/4036748174 >> 172.21.15.64:0/4036748174 conn(0x55cefa39a330 msgr2=0x55cefa3a7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 -- 172.21.15.64:0/4036748174 shutdown_connections 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 -- 172.21.15.64:0/4036748174 wait complete. 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.455+0000 7f96f3b93740 1 -- start start 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 1 Processor -- start 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1ab2a0 con 0x55cefa3a8f80 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1ac710 con 0x55cefa39dda0 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1ab970 con 0x55cefa39e6c0 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96f3b93740 10 client.0 _ll_register_callbacks cb 0x7f96d4010d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41776/0 (socket says 172.21.15.64:41776) 2024-08-19T01:41:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 -- 172.21.15.64:0/3319573569 learned_addr learned my addr 172.21.15.64:0/3319573569 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96eaffd640 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 -- 172.21.15.64:0/3319573569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 msgr2=0x55cefa3acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 -- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 msgr2=0x55cefa3b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.456+0000 7f96ea7fc640 1 -- 172.21.15.64:0/3319573569 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cefa3ad960 con 0x55cefa3a8f80 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96ea7fc640 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3ad130 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009c20 tx=0x7f96d4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae6ed45722b83d6 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96d27fc640 1 -- 172.21.15.64:0/3319573569 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4033040 con 0x55cefa3a8f80 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96d27fc640 1 -- 172.21.15.64:0/3319573569 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f96d4004050 con 0x55cefa3a8f80 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96d4005d30 con 0x55cefa3a8f80 2024-08-19T01:41:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96d27fc640 1 -- 172.21.15.64:0/3319573569 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f96d4043050 con 0x55cefa3a8f80 2024-08-19T01:41:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 10 client.15807 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cefa423db0 con 0x55cefa3a8f80 2024-08-19T01:41:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96d27fc640 1 -- 172.21.15.64:0/3319573569 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d4005d30 con 0x55cefa3a8f80 2024-08-19T01:41:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96d27fc640 1 -- 172.21.15.64:0/3319573569 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f96d4039070 con 0x55cefa3a8f80 2024-08-19T01:41:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 10 client.15807 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 10 client.15807 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 20 client.15807 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa424410 0x55cefa426800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 4 client.15807 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fcdc5a6a", "release": true, "await": true}] 2024-08-19T01:41:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.457+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fcdc5a6a", "release": true, "await": true}) -- 0x55cefa3076e0 con 0x55cefa424410 2024-08-19T01:41:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.458+0000 7f96eaffd640 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa424410 0x55cefa426800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.458+0000 7f96d27fc640 10 client.15807 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.458+0000 7f96eaffd640 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa424410 0x55cefa426800 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4184006b7c98112 server_cookie=ca5cd58b001f3b3e in_seq=0 out_seq=0 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.467+0000 7f96d27fc640 1 -- 172.21.15.64:0/3319573569 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cefa3076e0 con 0x55cefa424410 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.467+0000 7f96d27fc640 10 client.15807 handle_command_reply: tid=0 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.467+0000 7f96f3b93740 1 client.15807 shutdown 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 msgr2=0x55cefa3ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3ad130 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009c20 tx=0x7f96d4005740 comp rx=0 tx=0).stop 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 shutdown_connections 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cefa3a8f80 0x55cefa3ad130 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cefa39e6c0 0x55cefa3acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55cefa424410 0x55cefa426800 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 --2- 172.21.15.64:0/3319573569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cefa39dda0 0x55cefa3b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 >> 172.21.15.64:0/3319573569 conn(0x55cefa39a330 msgr2=0x55cefa394860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96d27fc640 0 client.15807 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 shutdown_connections 2024-08-19T01:41:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 1 -- 172.21.15.64:0/3319573569 wait complete. 2024-08-19T01:41:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f3b93740 20 client.15807 trim_cache size 0 max 16384 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 28, 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "fcdc5a6a": { 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 28, 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:41:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 143.8, 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:26.473 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:41:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f96d80a2cf0 msgr2=0x7f96d80a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f96d80a2cf0 0x7f96d80a51b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c6a0 tx=0x7f96dc097840 comp rx=0 tx=0).stop 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 msgr2=0x7f96ec19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec19ca40 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00e970 tx=0x7f96dc00ee30 comp rx=0 tx=0).stop 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 shutdown_connections 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f96d80a2cf0 0x7f96d80a51b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96ec1078a0 0x7f96ec19ca40 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96ec102240 0x7f96ec1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 --2- 172.21.15.64:0/494810983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96ec1018a0 0x7f96ec198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.468+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 >> 172.21.15.64:0/494810983 conn(0x7f96ec0fd5e0 msgr2=0x7f96ec0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.469+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 shutdown_connections 2024-08-19T01:41:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:26.469+0000 7f96f1795640 1 -- 172.21.15.64:0/494810983 wait complete. 2024-08-19T01:41:27.298 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'fcdc5a6a', total seconds elapsed: 27.0 2024-08-19T01:41:27.298 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.75 2024-08-19T01:41:27.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:41:27.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 728774 8.62 MB/sec execute 299 sec latency 23159.024 ms 2024-08-19T01:41:27.567 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:27.948 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:27 smithi064 ceph-mon[18515]: pgmap v253: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 2.6 MiB/s wr, 97 op/s 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.033+0000 7f237e2d6640 1 -- 172.21.15.64:0/661399746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 msgr2=0x7f237810e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.033+0000 7f237e2d6640 1 --2- 172.21.15.64:0/661399746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237810e970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2370009cd0 tx=0x7f237002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.033+0000 7f237e2d6640 1 -- 172.21.15.64:0/661399746 shutdown_connections 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.033+0000 7f237e2d6640 1 --2- 172.21.15.64:0/661399746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237810e970 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.033+0000 7f237e2d6640 1 --2- 172.21.15.64:0/661399746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2378103410 0x7f23781078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.033+0000 7f237e2d6640 1 --2- 172.21.15.64:0/661399746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378102a60 0x7f2378102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.033+0000 7f237e2d6640 1 -- 172.21.15.64:0/661399746 >> 172.21.15.64:0/661399746 conn(0x7f23780fc500 msgr2=0x7f23780fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.034+0000 7f237e2d6640 1 -- 172.21.15.64:0/661399746 shutdown_connections 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.034+0000 7f237e2d6640 1 -- 172.21.15.64:0/661399746 wait complete. 2024-08-19T01:41:28.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.034+0000 7f237e2d6640 1 Processor -- start 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.034+0000 7f237e2d6640 1 -- start start 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378102a60 0x7f2378198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2378103410 0x7f23781987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2378111dd0 con 0x7f2378107fd0 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2378111c50 con 0x7f2378102a60 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2378111f50 con 0x7f2378103410 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237c873640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f2377fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378102a60 0x7f2378198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237c873640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237819cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41802/0 (socket says 172.21.15.64:41802) 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237c873640 1 -- 172.21.15.64:0/758435284 learned_addr learned my addr 172.21.15.64:0/758435284 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f23777fe640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2378103410 0x7f23781987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f2377fff640 1 -- 172.21.15.64:0/758435284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2378103410 msgr2=0x7f23781987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:28.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f2377fff640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2378103410 0x7f23781987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f2377fff640 1 -- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 msgr2=0x7f237819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:28.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f2377fff640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f2377fff640 1 -- 172.21.15.64:0/758435284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237819d240 con 0x7f2378102a60 2024-08-19T01:41:28.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237c873640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:28.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f2377fff640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378102a60 0x7f2378198290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f236800e9f0 tx=0x7f236800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:28.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f23757fa640 1 -- 172.21.15.64:0/758435284 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236800ccc0 con 0x7f2378102a60 2024-08-19T01:41:28.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237819d530 con 0x7f2378102a60 2024-08-19T01:41:28.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f23757fa640 1 -- 172.21.15.64:0/758435284 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2368010910 con 0x7f2378102a60 2024-08-19T01:41:28.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f23757fa640 1 -- 172.21.15.64:0/758435284 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2368018960 con 0x7f2378102a60 2024-08-19T01:41:28.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.035+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23781a4e10 con 0x7f2378102a60 2024-08-19T01:41:28.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.036+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2378102e40 con 0x7f2378102a60 2024-08-19T01:41:28.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.038+0000 7f23757fa640 1 -- 172.21.15.64:0/758435284 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2368010ac0 con 0x7f2378102a60 2024-08-19T01:41:28.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.038+0000 7f23757fa640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f23600a2cf0 0x7f23600a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:28.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.038+0000 7f23757fa640 1 -- 172.21.15.64:0/758435284 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f23680cfef0 con 0x7f2378102a60 2024-08-19T01:41:28.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.038+0000 7f23777fe640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f23600a2cf0 0x7f23600a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:28.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.039+0000 7f23777fe640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f23600a2cf0 0x7f23600a51b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f23780feb30 tx=0x7f236c005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:28.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.042+0000 7f23757fa640 1 -- 172.21.15.64:0/758435284 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f236809c0b0 con 0x7f2378102a60 2024-08-19T01:41:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:27 smithi123 ceph-mon[21829]: pgmap v253: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 2.6 MiB/s wr, 97 op/s 2024-08-19T01:41:28.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:27 smithi161 ceph-mon[21757]: pgmap v253: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 2.6 MiB/s wr, 97 op/s 2024-08-19T01:41:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.306+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23781995f0 con 0x7f2378102a60 2024-08-19T01:41:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.307+0000 7f23757fa640 1 -- 172.21.15.64:0/758435284 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f23680a03b0 con 0x7f2378102a60 2024-08-19T01:41:28.308 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:41:28.308 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:41:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:41:28.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f23600a2cf0 msgr2=0x7f23600a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:28.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f23600a2cf0 0x7f23600a51b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f23780feb30 tx=0x7f236c005f90 comp rx=0 tx=0).stop 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378102a60 msgr2=0x7f2378198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378102a60 0x7f2378198290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f236800e9f0 tx=0x7f236800eeb0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 shutdown_connections 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f23600a2cf0 0x7f23600a51b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2378107fd0 0x7f237819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2378103410 0x7f23781987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 --2- 172.21.15.64:0/758435284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378102a60 0x7f2378198290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 >> 172.21.15.64:0/758435284 conn(0x7f23780fc500 msgr2=0x7f2378105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:28.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 shutdown_connections 2024-08-19T01:41:28.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:28.309+0000 7f237e2d6640 1 -- 172.21.15.64:0/758435284 wait complete. 2024-08-19T01:41:28.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 730959 8.63 MB/sec execute 300 sec latency 57.276 ms 2024-08-19T01:41:28.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:28 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/758435284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:29.031 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 143.75 --await-for 45.0' 2024-08-19T01:41:29.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 143.75 --await-for 45.0 2024-08-19T01:41:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:28 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/758435284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:29.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:28 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/758435284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:29.308 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:29.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 732746 8.63 MB/sec execute 301 sec latency 19.547 ms 2024-08-19T01:41:29.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.812+0000 7f8acb077640 1 -- 172.21.15.64:0/2447955574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4102c20 msgr2=0x7f8ac410d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.812+0000 7f8acb077640 1 --2- 172.21.15.64:0/2447955574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4102c20 0x7f8ac410d770 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009ca0 tx=0x7f8abc02a950 comp rx=0 tx=0).stop 2024-08-19T01:41:29.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.812+0000 7f8acb077640 1 -- 172.21.15.64:0/2447955574 shutdown_connections 2024-08-19T01:41:29.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.812+0000 7f8acb077640 1 --2- 172.21.15.64:0/2447955574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4102c20 0x7f8ac410d770 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.812+0000 7f8acb077640 1 --2- 172.21.15.64:0/2447955574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ac41022a0 0x7f8ac41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.812+0000 7f8acb077640 1 --2- 172.21.15.64:0/2447955574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac4100b50 0x7f8ac4100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.812+0000 7f8acb077640 1 -- 172.21.15.64:0/2447955574 >> 172.21.15.64:0/2447955574 conn(0x7f8ac40fc500 msgr2=0x7f8ac40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:29.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.813+0000 7f8acb077640 1 -- 172.21.15.64:0/2447955574 shutdown_connections 2024-08-19T01:41:29.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.813+0000 7f8acb077640 1 -- 172.21.15.64:0/2447955574 wait complete. 2024-08-19T01:41:29.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 Processor -- start 2024-08-19T01:41:29.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 -- start start 2024-08-19T01:41:29.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4100b50 0x7f8ac41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac41022a0 0x7f8ac4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ac4102c20 0x7f8ac419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac4110cb0 con 0x7f8ac4100b50 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac4110b30 con 0x7f8ac4102c20 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8acb077640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac4110e30 con 0x7f8ac41022a0 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8ac8e13640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4100b50 0x7f8ac41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8ac3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac41022a0 0x7f8ac4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8ac8e13640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4100b50 0x7f8ac41981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40932/0 (socket says 172.21.15.64:40932) 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8ac8e13640 1 -- 172.21.15.64:0/4079658352 learned_addr learned my addr 172.21.15.64:0/4079658352 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.814+0000 7f8ac9614640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ac4102c20 0x7f8ac419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac3fff640 1 -- 172.21.15.64:0/4079658352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ac4102c20 msgr2=0x7f8ac419ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ac4102c20 0x7f8ac419ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac3fff640 1 -- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4100b50 msgr2=0x7f8ac41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4100b50 0x7f8ac41981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac3fff640 1 -- 172.21.15.64:0/4079658352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac419d170 con 0x7f8ac41022a0 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4100b50 0x7f8ac41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac41022a0 0x7f8ac4198700 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000e9f0 tx=0x7f8ab000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac1ffb640 1 -- 172.21.15.64:0/4079658352 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab000ccc0 con 0x7f8ac41022a0 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac419d460 con 0x7f8ac41022a0 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac1ffb640 1 -- 172.21.15.64:0/4079658352 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab0010910 con 0x7f8ac41022a0 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.815+0000 7f8ac1ffb640 1 -- 172.21.15.64:0/4079658352 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0018960 con 0x7f8ac41022a0 2024-08-19T01:41:29.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.816+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac40fc8e0 con 0x7f8ac41022a0 2024-08-19T01:41:29.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 Processor -- start 2024-08-19T01:41:29.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8ac1ffb640 1 -- 172.21.15.64:0/4079658352 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8ab0010ac0 con 0x7f8ac41022a0 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 -- start start 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf76c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76cbe0 0x561fbf776f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf779890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561fbf6d1810 con 0x561fbf7774a0 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561fbf678c70 con 0x561fbf76c2c0 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8acd475740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561fbf6cd7d0 con 0x561fbf76cbe0 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8ac9614640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf76c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8ac9614640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf76c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:36218/0 (socket says 172.21.15.64:36218) 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.818+0000 7f8ac9614640 1 -- 172.21.15.64:0/773345435 learned_addr learned my addr 172.21.15.64:0/773345435 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76cbe0 0x561fbf776f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf779890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac9614640 1 -- 172.21.15.64:0/773345435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76cbe0 msgr2=0x561fbf776f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac9614640 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76cbe0 0x561fbf776f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac9614640 1 -- 172.21.15.64:0/773345435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 msgr2=0x561fbf779890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac9614640 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf779890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac9614640 1 -- 172.21.15.64:0/773345435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fbf443ba0 con 0x561fbf76c2c0 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf779890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac9614640 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf76c6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8abc035000 tx=0x7f8abc0093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a229c6b2c0c37aa0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac1ffb640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8ab40a2cf0 0x7f8ab40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8ab40a2cf0 0x7f8ab40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8aaeffd640 1 -- 172.21.15.64:0/773345435 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc048070 con 0x561fbf76c2c0 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8ac1ffb640 1 -- 172.21.15.64:0/4079658352 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f8ab00cfbf0 con 0x7f8ac41022a0 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8aaeffd640 1 -- 172.21.15.64:0/773345435 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8abc02f930 con 0x561fbf76c2c0 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.819+0000 7f8aaeffd640 1 -- 172.21.15.64:0/773345435 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc034040 con 0x561fbf76c2c0 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 -- 172.21.15.64:0/773345435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 msgr2=0x561fbf76c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8ab40a2cf0 0x7f8ab40a51b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x561fbf7696b0 tx=0x7f8ab8015040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf76c6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8abc035000 tx=0x7f8abc0093e0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 -- 172.21.15.64:0/773345435 shutdown_connections 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf779890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76cbe0 0x561fbf776f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 --2- 172.21.15.64:0/773345435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf76c6a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 -- 172.21.15.64:0/773345435 >> 172.21.15.64:0/773345435 conn(0x561fbf768850 msgr2=0x561fbf768c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 -- 172.21.15.64:0/773345435 shutdown_connections 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.820+0000 7f8acd475740 1 -- 172.21.15.64:0/773345435 wait complete. 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 -- start start 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 Processor -- start 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf774600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76cbe0 0x561fbf774b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf7750a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561fbf779fb0 con 0x561fbf7774a0 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561fbf68c9a0 con 0x561fbf76cbe0 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561fbf77a130 con 0x561fbf76c2c0 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8ac8e13640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf774600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8ac8e13640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf774600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:40394/0 (socket says 172.21.15.64:40394) 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8ac8e13640 1 -- 172.21.15.64:0/2297855792 learned_addr learned my addr 172.21.15.64:0/2297855792 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:29.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8acd475740 10 client.0 _ll_register_callbacks cb 0x561fbf67ac90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.821+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76cbe0 0x561fbf774b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac9614640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf7750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac8e13640 1 -- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76cbe0 msgr2=0x561fbf774b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76cbe0 0x561fbf774b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac8e13640 1 -- 172.21.15.64:0/2297855792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 msgr2=0x561fbf7750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf7750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac8e13640 1 -- 172.21.15.64:0/2297855792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fbf698220 con 0x561fbf76c2c0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76cbe0 0x561fbf774b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf774600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ab801fbe0 tx=0x7f8ab8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25f1ee2c44d006f0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8aa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8aa3fff640 1 -- 172.21.15.64:0/2297855792 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8025040 con 0x561fbf76c2c0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8abc02f930 con 0x561fbf76c2c0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8aa3fff640 1 -- 172.21.15.64:0/2297855792 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f8ab801d8c0 con 0x561fbf76c2c0 2024-08-19T01:41:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8aa3fff640 1 -- 172.21.15.64:0/2297855792 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8ab8037050 con 0x561fbf76c2c0 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8acd475740 10 client.25267 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.822+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561fbf785e70 con 0x561fbf76c2c0 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8aa3fff640 1 -- 172.21.15.64:0/2297855792 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8abc02f930 con 0x561fbf76c2c0 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8aa3fff640 1 -- 172.21.15.64:0/2297855792 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8ab8029cb0 con 0x561fbf76c2c0 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8acd475740 10 client.25267 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8acd475740 10 client.25267 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8acd475740 20 client.25267 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8acd475740 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf787330 0x561fbf789760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8acd475740 4 client.25267 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fbf6def40 con 0x561fbf787330 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.823+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf787330 0x561fbf789760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf787330 0x561fbf789760 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0cbaf4b3596bece server_cookie=eb05c932e5a504f4 in_seq=0 out_seq=0 2024-08-19T01:41:29.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8aa3fff640 10 client.25267 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8aa3fff640 1 -- 172.21.15.64:0/2297855792 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561fbf6def40 con 0x561fbf787330 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8aa3fff640 10 client.25267 handle_command_reply: tid=0 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 client.25267 shutdown 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 msgr2=0x561fbf774600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf774600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ab801fbe0 tx=0x7f8ab8004730 comp rx=0 tx=0).stop 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 shutdown_connections 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf7774a0 0x561fbf7750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf787330 0x561fbf789760 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76cbe0 0x561fbf774b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 --2- 172.21.15.64:0/2297855792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf774600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 >> 172.21.15.64:0/2297855792 conn(0x561fbf768850 msgr2=0x561fbf767b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.824+0000 7f8aa3fff640 0 client.25267 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.825+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 shutdown_connections 2024-08-19T01:41:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.825+0000 7f8acd475740 1 -- 172.21.15.64:0/2297855792 wait complete. 2024-08-19T01:41:29.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.825+0000 7f8acd475740 20 client.25267 trim_cache size 0 max 16384 2024-08-19T01:41:29.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 Processor -- start 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 -- start start 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf78f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf78f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf7774a0 0x561fbf77f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac41af4a0 con 0x561fbf76cbe0 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac41acd70 con 0x561fbf7774a0 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8acd475740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac41aabe0 con 0x561fbf76c2c0 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8ac3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf78f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8ac3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf78f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40958/0 (socket says 172.21.15.64:40958) 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8ac3fff640 1 -- 172.21.15.64:0/3780053793 learned_addr learned my addr 172.21.15.64:0/3780053793 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8ac9614640 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf7774a0 0x561fbf77f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.843+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf78f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8ac3fff640 1 -- 172.21.15.64:0/3780053793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 msgr2=0x561fbf78f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf78f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8ac3fff640 1 -- 172.21.15.64:0/3780053793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf7774a0 msgr2=0x561fbf77f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf7774a0 0x561fbf77f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8ac3fff640 1 -- 172.21.15.64:0/3780053793 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fbf779bc0 con 0x561fbf76cbe0 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8ac9614640 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf7774a0 0x561fbf77f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8ac3fff640 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf78f590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000e9c0 tx=0x7f8ab0009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b579666e9484eec7 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8aae7fc640 1 -- 172.21.15.64:0/3780053793 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00a4970 con 0x561fbf76cbe0 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8aae7fc640 1 -- 172.21.15.64:0/3780053793 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab009a0b0 con 0x561fbf76cbe0 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.844+0000 7f8aae7fc640 1 -- 172.21.15.64:0/3780053793 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0018b00 con 0x561fbf76cbe0 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 -- 172.21.15.64:0/3780053793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 msgr2=0x561fbf78f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf78f590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000e9c0 tx=0x7f8ab0009700 comp rx=0 tx=0).stop 2024-08-19T01:41:29.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 -- 172.21.15.64:0/3780053793 shutdown_connections 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf7774a0 0x561fbf77f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf78f590 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 --2- 172.21.15.64:0/3780053793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf76c2c0 0x561fbf78f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 -- 172.21.15.64:0/3780053793 >> 172.21.15.64:0/3780053793 conn(0x561fbf768850 msgr2=0x561fbf772f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 -- 172.21.15.64:0/3780053793 shutdown_connections 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.845+0000 7f8acd475740 1 -- 172.21.15.64:0/3780053793 wait complete. 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 -- start start 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 Processor -- start 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf77f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf771500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf7774a0 0x561fbf771a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac41ab230 con 0x561fbf76cbe0 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac41ac6a0 con 0x561fbf76c2c0 2024-08-19T01:41:29.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac41ab900 con 0x561fbf7774a0 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8acd475740 10 client.0 _ll_register_callbacks cb 0x7f8ab8022040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.846+0000 7f8ac3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf771500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac9614640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf7774a0 0x561fbf771a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac8e13640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf77f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf771500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40966/0 (socket says 172.21.15.64:40966) 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac3fff640 1 -- 172.21.15.64:0/229906537 learned_addr learned my addr 172.21.15.64:0/229906537 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac9614640 1 -- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 msgr2=0x561fbf77f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac9614640 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf77f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac9614640 1 -- 172.21.15.64:0/229906537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 msgr2=0x561fbf771500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac9614640 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf771500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac9614640 1 -- 172.21.15.64:0/229906537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fbf772080 con 0x561fbf7774a0 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf77f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8ac9614640 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf7774a0 0x561fbf771a40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009ca0 tx=0x7f8abc003c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da78c49d7388de9d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:29.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8aa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8aa3fff640 1 -- 172.21.15.64:0/229906537 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc048040 con 0x561fbf7774a0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8aa3fff640 1 -- 172.21.15.64:0/229906537 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f8abc02f5d0 con 0x561fbf7774a0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.847+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ab009a0b0 con 0x561fbf7774a0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8aa3fff640 1 -- 172.21.15.64:0/229906537 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8abc05d050 con 0x561fbf7774a0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 10 client.25273 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561fbf7f1e50 con 0x561fbf7774a0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8aa3fff640 1 -- 172.21.15.64:0/229906537 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab009a0b0 con 0x561fbf7774a0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8aa3fff640 1 -- 172.21.15.64:0/229906537 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8abc030440 con 0x561fbf7774a0 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 10 client.25273 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 10 client.25273 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 20 client.25273 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf7f2190 0x561fbf7f4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 4 client.25273 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.75, "await_for": 45.0}] 2024-08-19T01:41:29.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.75, "await_for": 45.0}) -- 0x561fbf6d9290 con 0x561fbf7f2190 2024-08-19T01:41:29.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.848+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf7f2190 0x561fbf7f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:29.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.849+0000 7f8ac8e13640 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf7f2190 0x561fbf7f4580 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aee5b1da187c27c server_cookie=d4129d6d53f3f42d in_seq=0 out_seq=0 2024-08-19T01:41:29.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.849+0000 7f8aa3fff640 10 client.25273 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:29.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8aa3fff640 1 -- 172.21.15.64:0/229906537 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561fbf6d9290 con 0x561fbf7f2190 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8aa3fff640 10 client.25273 handle_command_reply: tid=0 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 client.25273 shutdown 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf7774a0 msgr2=0x561fbf771a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf7774a0 0x561fbf771a40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009ca0 tx=0x7f8abc003c30 comp rx=0 tx=0).stop 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 shutdown_connections 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561fbf7774a0 0x561fbf771a40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561fbf76cbe0 0x561fbf771500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561fbf7f2190 0x561fbf7f4580 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 --2- 172.21.15.64:0/229906537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561fbf76c2c0 0x561fbf77f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8aa3fff640 0 client.25273 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:29.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 >> 172.21.15.64:0/229906537 conn(0x561fbf768850 msgr2=0x561fbf767310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:29.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 shutdown_connections 2024-08-19T01:41:29.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 1 -- 172.21.15.64:0/229906537 wait complete. 2024-08-19T01:41:29.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.917+0000 7f8acd475740 20 client.25273 trim_cache size 0 max 16384 2024-08-19T01:41:29.920 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 30, 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "5c5b6f92": { 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 30, 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:41:29.921 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 143.8, 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:29.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:29.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:29.923 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:41:29.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8ab40a2cf0 msgr2=0x7f8ab40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8ab40a2cf0 0x7f8ab40a51b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x561fbf7696b0 tx=0x7f8ab8015040 comp rx=0 tx=0).stop 2024-08-19T01:41:29.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac41022a0 msgr2=0x7f8ac4198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:29.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac41022a0 0x7f8ac4198700 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000e9f0 tx=0x7f8ab000eeb0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 shutdown_connections 2024-08-19T01:41:29.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8ab40a2cf0 0x7f8ab40a51b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ac4102c20 0x7f8ac419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac41022a0 0x7f8ac4198700 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 --2- 172.21.15.64:0/4079658352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ac4100b50 0x7f8ac41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:29.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 >> 172.21.15.64:0/4079658352 conn(0x7f8ac40fc500 msgr2=0x7f8ac4069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:29.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 shutdown_connections 2024-08-19T01:41:29.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:29.918+0000 7f8acb077640 1 -- 172.21.15.64:0/4079658352 wait complete. 2024-08-19T01:41:29.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:29 smithi064 ceph-mon[18515]: pgmap v254: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 1.8 MiB/s wr, 66 op/s 2024-08-19T01:41:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:29 smithi123 ceph-mon[21829]: pgmap v254: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 1.8 MiB/s wr, 66 op/s 2024-08-19T01:41:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:29 smithi161 ceph-mon[21757]: pgmap v254: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 1.8 MiB/s wr, 66 op/s 2024-08-19T01:41:30.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.62 MB/sec execute 302 sec latency 480.545 ms 2024-08-19T01:41:30.735 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5c5b6f92', quiesce took 3.4 seconds. Will release after: 20.35 2024-08-19T01:41:31.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.59 MB/sec execute 303 sec latency 1480.629 ms 2024-08-19T01:41:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:31 smithi123 ceph-mon[21829]: pgmap v255: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 156 op/s 2024-08-19T01:41:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:31 smithi161 ceph-mon[21757]: pgmap v255: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 156 op/s 2024-08-19T01:41:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:31 smithi064 ceph-mon[18515]: pgmap v255: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 156 op/s 2024-08-19T01:41:32.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.56 MB/sec execute 304 sec latency 2480.732 ms 2024-08-19T01:41:33.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.53 MB/sec execute 305 sec latency 3480.819 ms 2024-08-19T01:41:34.001 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:33 smithi161 ceph-mon[21757]: pgmap v256: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 117 op/s 2024-08-19T01:41:34.037 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:33 smithi064 ceph-mon[18515]: pgmap v256: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 117 op/s 2024-08-19T01:41:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:33 smithi123 ceph-mon[21829]: pgmap v256: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 117 op/s 2024-08-19T01:41:34.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.51 MB/sec execute 306 sec latency 4480.879 ms 2024-08-19T01:41:35.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.48 MB/sec execute 307 sec latency 5480.965 ms 2024-08-19T01:41:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:35 smithi123 ceph-mon[21829]: pgmap v257: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 117 op/s 2024-08-19T01:41:36.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:35 smithi161 ceph-mon[21757]: pgmap v257: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 117 op/s 2024-08-19T01:41:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:35 smithi064 ceph-mon[18515]: pgmap v257: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 117 op/s 2024-08-19T01:41:36.398 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.45 MB/sec execute 308 sec latency 6481.059 ms 2024-08-19T01:41:37.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.42 MB/sec execute 309 sec latency 7481.172 ms 2024-08-19T01:41:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:37 smithi123 ceph-mon[21829]: pgmap v258: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 134 op/s 2024-08-19T01:41:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:37 smithi161 ceph-mon[21757]: pgmap v258: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 134 op/s 2024-08-19T01:41:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:37 smithi064 ceph-mon[18515]: pgmap v258: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 134 op/s 2024-08-19T01:41:38.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.40 MB/sec execute 310 sec latency 8481.263 ms 2024-08-19T01:41:39.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.37 MB/sec execute 311 sec latency 9481.321 ms 2024-08-19T01:41:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:39 smithi123 ceph-mon[21829]: pgmap v259: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-08-19T01:41:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:39 smithi161 ceph-mon[21757]: pgmap v259: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-08-19T01:41:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:39 smithi064 ceph-mon[18515]: pgmap v259: 129 pgs: 129 active+clean; 443 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-08-19T01:41:40.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.34 MB/sec execute 312 sec latency 10481.408 ms 2024-08-19T01:41:41.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.31 MB/sec execute 313 sec latency 11481.469 ms 2024-08-19T01:41:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:41 smithi123 ceph-mon[21829]: pgmap v260: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 123 op/s 2024-08-19T01:41:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:41 smithi161 ceph-mon[21757]: pgmap v260: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 123 op/s 2024-08-19T01:41:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:41 smithi064 ceph-mon[18515]: pgmap v260: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 123 op/s 2024-08-19T01:41:42.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.29 MB/sec execute 314 sec latency 12481.530 ms 2024-08-19T01:41:43.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.26 MB/sec execute 315 sec latency 13481.569 ms 2024-08-19T01:41:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:43 smithi123 ceph-mon[21829]: pgmap v261: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 615 KiB/s wr, 33 op/s 2024-08-19T01:41:44.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:43 smithi161 ceph-mon[21757]: pgmap v261: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 615 KiB/s wr, 33 op/s 2024-08-19T01:41:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:43 smithi064 ceph-mon[18515]: pgmap v261: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 615 KiB/s wr, 33 op/s 2024-08-19T01:41:44.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.24 MB/sec execute 316 sec latency 14481.676 ms 2024-08-19T01:41:45.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.21 MB/sec execute 317 sec latency 15481.764 ms 2024-08-19T01:41:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:45 smithi123 ceph-mon[21829]: pgmap v262: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 615 KiB/s wr, 33 op/s 2024-08-19T01:41:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:45 smithi161 ceph-mon[21757]: pgmap v262: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 615 KiB/s wr, 33 op/s 2024-08-19T01:41:46.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:45 smithi064 ceph-mon[18515]: pgmap v262: 129 pgs: 129 active+clean; 438 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 615 KiB/s wr, 33 op/s 2024-08-19T01:41:46.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.18 MB/sec execute 318 sec latency 16481.867 ms 2024-08-19T01:41:47.399 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.16 MB/sec execute 319 sec latency 17481.972 ms 2024-08-19T01:41:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:47 smithi123 ceph-mon[21829]: pgmap v263: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 634 KiB/s wr, 36 op/s 2024-08-19T01:41:48.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:47 smithi161 ceph-mon[21757]: pgmap v263: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 634 KiB/s wr, 36 op/s 2024-08-19T01:41:48.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:47 smithi064 ceph-mon[18515]: pgmap v263: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 634 KiB/s wr, 36 op/s 2024-08-19T01:41:48.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.13 MB/sec execute 320 sec latency 18482.069 ms 2024-08-19T01:41:49.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.11 MB/sec execute 321 sec latency 19482.132 ms 2024-08-19T01:41:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:49 smithi123 ceph-mon[21829]: pgmap v264: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 142 KiB/s wr, 19 op/s 2024-08-19T01:41:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:49 smithi161 ceph-mon[21757]: pgmap v264: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 142 KiB/s wr, 19 op/s 2024-08-19T01:41:50.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:49 smithi064 ceph-mon[18515]: pgmap v264: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 142 KiB/s wr, 19 op/s 2024-08-19T01:41:50.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.08 MB/sec execute 322 sec latency 20482.189 ms 2024-08-19T01:41:51.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:41:51.362 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:51.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.06 MB/sec execute 323 sec latency 21482.240 ms 2024-08-19T01:41:51.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 -- 172.21.15.64:0/2355185938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc102b00 msgr2=0x7f6cfc102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:51.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2355185938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc102b00 0x7f6cfc102ee0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009930 tx=0x7f6cec02a420 comp rx=0 tx=0).stop 2024-08-19T01:41:51.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 -- 172.21.15.64:0/2355185938 shutdown_connections 2024-08-19T01:41:51.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2355185938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc108030 0x7f6cfc10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:51.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2355185938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cfc1034b0 0x7f6cfc1078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:51.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2355185938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc102b00 0x7f6cfc102ee0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 -- 172.21.15.64:0/2355185938 >> 172.21.15.64:0/2355185938 conn(0x7f6cfc0fc500 msgr2=0x7f6cfc0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 -- 172.21.15.64:0/2355185938 shutdown_connections 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 -- 172.21.15.64:0/2355185938 wait complete. 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.876+0000 7f6d015ea640 1 Processor -- start 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6d015ea640 1 -- start start 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6d015ea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cfc102b00 0x7f6cfc100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6d015ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc1034b0 0x7f6cfc100ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6d015ea640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc108030 0x7f6cfc1011e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6d015ea640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc111e40 con 0x7f6cfc108030 2024-08-19T01:41:51.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6d015ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc111cc0 con 0x7f6cfc1034b0 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6d015ea640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc111fc0 con 0x7f6cfc102b00 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc108030 0x7f6cfc1011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc108030 0x7f6cfc1011e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46026/0 (socket says 172.21.15.64:46026) 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 -- 172.21.15.64:0/2158769276 learned_addr learned my addr 172.21.15.64:0/2158769276 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cf3fff640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc1034b0 0x7f6cfc100ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfaffd640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cfc102b00 0x7f6cfc100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 -- 172.21.15.64:0/2158769276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cfc102b00 msgr2=0x7f6cfc100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cfc102b00 0x7f6cfc100760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 -- 172.21.15.64:0/2158769276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc1034b0 msgr2=0x7f6cfc100ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc1034b0 0x7f6cfc100ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:51.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 -- 172.21.15.64:0/2158769276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc1a6800 con 0x7f6cfc108030 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.877+0000 7f6cfb7fe640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc108030 0x7f6cfc1011e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400b970 tx=0x7f6cf400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.878+0000 7f6cf8ff9640 1 -- 172.21.15.64:0/2158769276 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4019070 con 0x7f6cfc108030 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.878+0000 7f6cf8ff9640 1 -- 172.21.15.64:0/2158769276 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf400cb40 con 0x7f6cfc108030 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.878+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc1a6af0 con 0x7f6cfc108030 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.878+0000 7f6cf8ff9640 1 -- 172.21.15.64:0/2158769276 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4007650 con 0x7f6cfc108030 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.878+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc1a6fe0 con 0x7f6cfc108030 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.879+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cfc06a4e0 con 0x7f6cfc108030 2024-08-19T01:41:51.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.880+0000 7f6cf8ff9640 1 -- 172.21.15.64:0/2158769276 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6cf401c070 con 0x7f6cfc108030 2024-08-19T01:41:51.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.880+0000 7f6cf8ff9640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f6ce80a2cf0 0x7f6ce80a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:51.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.881+0000 7f6cfaffd640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f6ce80a2cf0 0x7f6ce80a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:51.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.881+0000 7f6cf8ff9640 1 -- 172.21.15.64:0/2158769276 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f6cf40cecf0 con 0x7f6cfc108030 2024-08-19T01:41:51.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.881+0000 7f6cfaffd640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f6ce80a2cf0 0x7f6ce80a51b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6cec000c00 tx=0x7f6cec005a20 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:51.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:51.885+0000 7f6cf8ff9640 1 -- 172.21.15.64:0/2158769276 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f6cf40d3070 con 0x7f6cfc108030 2024-08-19T01:41:52.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:51 smithi064 ceph-mon[18515]: pgmap v265: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 147 KiB/s wr, 22 op/s 2024-08-19T01:41:52.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.097+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cfc10a010 con 0x7f6cfc108030 2024-08-19T01:41:52.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.098+0000 7f6cf8ff9640 1 -- 172.21.15.64:0/2158769276 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f6cf409afc0 con 0x7f6cfc108030 2024-08-19T01:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:41:52.102 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f6ce80a2cf0 msgr2=0x7f6ce80a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f6ce80a2cf0 0x7f6ce80a51b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6cec000c00 tx=0x7f6cec005a20 comp rx=0 tx=0).stop 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc108030 msgr2=0x7f6cfc1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc108030 0x7f6cfc1011e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400b970 tx=0x7f6cf400be30 comp rx=0 tx=0).stop 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 shutdown_connections 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f6ce80a2cf0 0x7f6ce80a51b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cfc108030 0x7f6cfc1011e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc1034b0 0x7f6cfc100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 --2- 172.21.15.64:0/2158769276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cfc102b00 0x7f6cfc100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 >> 172.21.15.64:0/2158769276 conn(0x7f6cfc0fc500 msgr2=0x7f6cfc1060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:52.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.100+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 shutdown_connections 2024-08-19T01:41:52.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:52.101+0000 7f6d015ea640 1 -- 172.21.15.64:0/2158769276 wait complete. 2024-08-19T01:41:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:51 smithi123 ceph-mon[21829]: pgmap v265: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 147 KiB/s wr, 22 op/s 2024-08-19T01:41:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:51 smithi161 ceph-mon[21757]: pgmap v265: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 147 KiB/s wr, 22 op/s 2024-08-19T01:41:52.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.03 MB/sec execute 324 sec latency 22482.299 ms 2024-08-19T01:41:52.853 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 5c5b6f92 --release --await' 2024-08-19T01:41:52.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 5c5b6f92 --release --await 2024-08-19T01:41:53.111 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:41:53.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:52 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2158769276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:53.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:52 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2158769276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:53.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:52 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2158769276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:41:53.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 733839 8.01 MB/sec execute 325 sec latency 23482.330 ms 2024-08-19T01:41:53.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 -- 172.21.15.64:0/2516730928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174102c20 msgr2=0x7fe17410d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 --2- 172.21.15.64:0/2516730928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174102c20 0x7fe17410d770 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009ca0 tx=0x7fe16c02a950 comp rx=0 tx=0).stop 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 -- 172.21.15.64:0/2516730928 shutdown_connections 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 --2- 172.21.15.64:0/2516730928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174102c20 0x7fe17410d770 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 --2- 172.21.15.64:0/2516730928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1741022a0 0x7fe1741026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 --2- 172.21.15.64:0/2516730928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe174100b50 0x7fe174100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 -- 172.21.15.64:0/2516730928 >> 172.21.15.64:0/2516730928 conn(0x7fe1740fc500 msgr2=0x7fe1740fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 -- 172.21.15.64:0/2516730928 shutdown_connections 2024-08-19T01:41:53.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.618+0000 7fe17aa06640 1 -- 172.21.15.64:0/2516730928 wait complete. 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 Processor -- start 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 -- start start 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174100b50 0x7fe174198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1741022a0 0x7fe1741986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe174102c20 0x7fe17419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe174111dd0 con 0x7fe174100b50 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe174111c50 con 0x7fe174102c20 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe17aa06640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe174111f50 con 0x7fe1741022a0 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe173fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174100b50 0x7fe174198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe173fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174100b50 0x7fe174198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46054/0 (socket says 172.21.15.64:46054) 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe173fff640 1 -- 172.21.15.64:0/1620900240 learned_addr learned my addr 172.21.15.64:0/1620900240 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe178fa3640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe174102c20 0x7fe17419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe16bfff640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1741022a0 0x7fe1741986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe173fff640 1 -- 172.21.15.64:0/1620900240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1741022a0 msgr2=0x7fe1741986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe173fff640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1741022a0 0x7fe1741986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe173fff640 1 -- 172.21.15.64:0/1620900240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe174102c20 msgr2=0x7fe17419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.619+0000 7fe173fff640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe174102c20 0x7fe17419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe173fff640 1 -- 172.21.15.64:0/1620900240 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17419d120 con 0x7fe174100b50 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe16bfff640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1741022a0 0x7fe1741986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe173fff640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174100b50 0x7fe174198170 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe16400d950 tx=0x7fe16400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe171ffb640 1 -- 172.21.15.64:0/1620900240 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164017030 con 0x7fe174100b50 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe171ffb640 1 -- 172.21.15.64:0/1620900240 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe164004020 con 0x7fe174100b50 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17419d410 con 0x7fe174100b50 2024-08-19T01:41:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe171ffb640 1 -- 172.21.15.64:0/1620900240 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1640047a0 con 0x7fe174100b50 2024-08-19T01:41:53.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.620+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1741a4cf0 con 0x7fe174100b50 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.622+0000 7fe171ffb640 1 -- 172.21.15.64:0/1620900240 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe1640041d0 con 0x7fe174100b50 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.622+0000 7fe17ce04740 1 Processor -- start 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe17ce04740 1 -- start start 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fb990 0x5591e86fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8706630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e8706b70 0x5591e8708f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe17ce04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5591e8661c60 con 0x5591e8706b70 2024-08-19T01:41:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe17ce04740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5591e85f9b90 con 0x5591e86fc2b0 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe17ce04740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5591e865dc20 con 0x5591e86fb990 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e8706b70 0x5591e8708f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e8706b70 0x5591e8708f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46064/0 (socket says 172.21.15.64:46064) 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 -- 172.21.15.64:0/3873477271 learned_addr learned my addr 172.21.15.64:0/3873477271 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe171ffb640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe1600a2cf0 0x7fe1600a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe173fff640 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8706630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe178fa3640 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fb990 0x5591e86fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 -- 172.21.15.64:0/3873477271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fb990 msgr2=0x5591e86fbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fb990 0x5591e86fbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 -- 172.21.15.64:0/3873477271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 msgr2=0x5591e8706630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8706630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe16bfff640 1 -- 172.21.15.64:0/3873477271 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591e8371580 con 0x5591e8706b70 2024-08-19T01:41:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe171ffb640 1 -- 172.21.15.64:0/1620900240 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fe1640cf330 con 0x7fe174100b50 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.623+0000 7fe178fa3640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe1600a2cf0 0x7fe1600a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.624+0000 7fe16bfff640 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e8706b70 0x5591e8708f60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009b90 tx=0x7fe15c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d763c1794eda870c server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.624+0000 7fe16a7fc640 1 -- 172.21.15.64:0/3873477271 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c016070 con 0x5591e8706b70 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.624+0000 7fe16a7fc640 1 -- 172.21.15.64:0/3873477271 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe15c005500 con 0x5591e8706b70 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.624+0000 7fe16a7fc640 1 -- 172.21.15.64:0/3873477271 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c011470 con 0x5591e8706b70 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.624+0000 7fe17ce04740 1 -- 172.21.15.64:0/3873477271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e8706b70 msgr2=0x5591e8708f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.624+0000 7fe178fa3640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe1600a2cf0 0x7fe1600a51b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe16c005e30 tx=0x7fe16c0095c0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.624+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e8706b70 0x5591e8708f60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009b90 tx=0x7fe15c00e6c0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 -- 172.21.15.64:0/3873477271 shutdown_connections 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e8706b70 0x5591e8708f60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8706630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3873477271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fb990 0x5591e86fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 -- 172.21.15.64:0/3873477271 >> 172.21.15.64:0/3873477271 conn(0x5591e86f7f20 msgr2=0x5591e86f8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 -- 172.21.15.64:0/3873477271 shutdown_connections 2024-08-19T01:41:53.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 -- 172.21.15.64:0/3873477271 wait complete. 2024-08-19T01:41:53.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 -- start start 2024-08-19T01:41:53.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 Processor -- start 2024-08-19T01:41:53.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e87039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fc2b0 0x5591e8703f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.625+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e8706b70 0x5591e8704450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe17ce04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5591e870db50 con 0x5591e86fb990 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe17ce04740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5591e85ff7c0 con 0x5591e8706b70 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe17ce04740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5591e870dcd0 con 0x5591e86fc2b0 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe17ce04740 10 client.0 _ll_register_callbacks cb 0x5591e8608f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e87039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e87039d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46078/0 (socket says 172.21.15.64:46078) 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 -- 172.21.15.64:0/3690299006 learned_addr learned my addr 172.21.15.64:0/3690299006 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe16bfff640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fc2b0 0x5591e8703f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe178fa3640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e8706b70 0x5591e8704450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 -- 172.21.15.64:0/3690299006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fc2b0 msgr2=0x5591e8703f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fc2b0 0x5591e8703f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 -- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e8706b70 msgr2=0x5591e8704450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e8706b70 0x5591e8704450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 -- 172.21.15.64:0/3690299006 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591e843b960 con 0x5591e86fb990 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe178fa3640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e8706b70 0x5591e8704450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe173fff640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e87039d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe1640cc670 tx=0x7fe16409afe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b0bb693300a82b server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe1537fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe1537fe640 1 -- 172.21.15.64:0/3690299006 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16409f790 con 0x5591e86fb990 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe1537fe640 1 -- 172.21.15.64:0/3690299006 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fe1640cc890 con 0x5591e86fb990 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.626+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe15c005500 con 0x5591e86fb990 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe1537fe640 1 -- 172.21.15.64:0/3690299006 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe1640cca90 con 0x5591e86fb990 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 10 client.15849 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:53.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5591e8719b10 con 0x5591e86fb990 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe1537fe640 1 -- 172.21.15.64:0/3690299006 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe15c005500 con 0x5591e86fb990 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe1537fe640 1 -- 172.21.15.64:0/3690299006 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe1640b4260 con 0x5591e86fb990 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 10 client.15849 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 10 client.15849 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 20 client.15849 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e871ae60 0x5591e871d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 4 client.15849 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591e866f1d0 con 0x5591e871ae60 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.627+0000 7fe16bfff640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e871ae60 0x5591e871d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.628+0000 7fe16bfff640 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e871ae60 0x5591e871d270 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d53efb18deee37 server_cookie=7de7907163db9eb3 in_seq=0 out_seq=0 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.628+0000 7fe1537fe640 10 client.15849 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe1537fe640 1 -- 172.21.15.64:0/3690299006 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591e866f1d0 con 0x5591e871ae60 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe1537fe640 10 client.15849 handle_command_reply: tid=0 2024-08-19T01:41:53.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 client.15849 shutdown 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 msgr2=0x5591e87039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e87039d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe1640cc670 tx=0x7fe16409afe0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 shutdown_connections 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e871ae60 0x5591e871d270 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e8706b70 0x5591e8704450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e86fc2b0 0x5591e8703f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 --2- 172.21.15.64:0/3690299006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e87039d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe1537fe640 0 client.15849 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 >> 172.21.15.64:0/3690299006 conn(0x5591e86f7f20 msgr2=0x5591e86fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 shutdown_connections 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 1 -- 172.21.15.64:0/3690299006 wait complete. 2024-08-19T01:41:53.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.629+0000 7fe17ce04740 20 client.15849 trim_cache size 0 max 16384 2024-08-19T01:41:53.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 Processor -- start 2024-08-19T01:41:53.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 -- start start 2024-08-19T01:41:53.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e8705940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8722b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e8723080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe1741af520 con 0x5591e86fb990 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe1741acdf0 con 0x5591e86fc2b0 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe17ce04740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe1741aac60 con 0x5591e8706b70 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe173fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e8705940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe173fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e8705940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46094/0 (socket says 172.21.15.64:46094) 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.646+0000 7fe173fff640 1 -- 172.21.15.64:0/2486410020 learned_addr learned my addr 172.21.15.64:0/2486410020 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe178fa3640 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e8723080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe16bfff640 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8722b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe173fff640 1 -- 172.21.15.64:0/2486410020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 msgr2=0x5591e8723080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe173fff640 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e8723080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe173fff640 1 -- 172.21.15.64:0/2486410020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 msgr2=0x5591e8722b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe173fff640 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8722b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe173fff640 1 -- 172.21.15.64:0/2486410020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591e86f7d50 con 0x5591e86fb990 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe178fa3640 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e8723080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe173fff640 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e8705940 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe164009610 tx=0x7fe1640998f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc076648c3b7bf2 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe169ffb640 1 -- 172.21.15.64:0/2486410020 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1640a4050 con 0x5591e86fb990 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe169ffb640 1 -- 172.21.15.64:0/2486410020 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1640b9020 con 0x5591e86fb990 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.647+0000 7fe169ffb640 1 -- 172.21.15.64:0/2486410020 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1640b4340 con 0x5591e86fb990 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 -- 172.21.15.64:0/2486410020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 msgr2=0x5591e8705940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e8705940 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe164009610 tx=0x7fe1640998f0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 -- 172.21.15.64:0/2486410020 shutdown_connections 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e8723080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fc2b0 0x5591e8722b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 --2- 172.21.15.64:0/2486410020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fb990 0x5591e8705940 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 -- 172.21.15.64:0/2486410020 >> 172.21.15.64:0/2486410020 conn(0x5591e86f7f20 msgr2=0x5591e8705230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 -- 172.21.15.64:0/2486410020 shutdown_connections 2024-08-19T01:41:53.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 -- 172.21.15.64:0/2486410020 wait complete. 2024-08-19T01:41:53.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 -- start start 2024-08-19T01:41:53.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.648+0000 7fe17ce04740 1 Processor -- start 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fb990 0x5591e8712f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fc2b0 0x5591e8713460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e870fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe1741ab2b0 con 0x5591e86fc2b0 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe1741ac720 con 0x5591e86fb990 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe1741ab980 con 0x5591e8706b70 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 10 client.0 _ll_register_callbacks cb 0x7fe164005130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fc2b0 0x5591e8713460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fc2b0 0x5591e8713460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46108/0 (socket says 172.21.15.64:46108) 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 -- 172.21.15.64:0/9111868 learned_addr learned my addr 172.21.15.64:0/9111868 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe173fff640 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fb990 0x5591e8712f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe178fa3640 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e870fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 -- 172.21.15.64:0/9111868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 msgr2=0x5591e870fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e870fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 -- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fb990 msgr2=0x5591e8712f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fb990 0x5591e8712f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 -- 172.21.15.64:0/9111868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591e8710530 con 0x5591e86fc2b0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe16bfff640 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fc2b0 0x5591e8713460 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009780 tx=0x7fe15c0092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9f62eeabb3547c server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe1537fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe1537fe640 1 -- 172.21.15.64:0/9111868 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c016040 con 0x5591e86fc2b0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe1537fe640 1 -- 172.21.15.64:0/9111868 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fe15c011470 con 0x5591e86fc2b0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1640b9020 con 0x5591e86fc2b0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe1537fe640 1 -- 172.21.15.64:0/9111868 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe15c031680 con 0x5591e86fc2b0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 10 client.15855 fetch_fsmap learned FSMap version 29 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.649+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5591e8786460 con 0x5591e86fc2b0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe1537fe640 1 -- 172.21.15.64:0/9111868 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1640b9020 con 0x5591e86fc2b0 2024-08-19T01:41:53.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe1537fe640 1 -- 172.21.15.64:0/9111868 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe15c00f040 con 0x5591e86fc2b0 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe17ce04740 10 client.15855 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe17ce04740 10 client.15855 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe17ce04740 20 client.15855 populate_metadata read hostname 'smithi064' 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe17ce04740 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e8786ad0 0x5591e8788ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe17ce04740 4 client.15855 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5c5b6f92", "release": true, "await": true}] 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5c5b6f92", "release": true, "await": true}) -- 0x5591e86696e0 con 0x5591e8786ad0 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe173fff640 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e8786ad0 0x5591e8788ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe173fff640 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e8786ad0 0x5591e8788ef0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=310669be73f7c227 server_cookie=7065b9bd8b7564d0 in_seq=0 out_seq=0 2024-08-19T01:41:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.650+0000 7fe1537fe640 10 client.15855 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.673+0000 7fe1537fe640 1 -- 172.21.15.64:0/9111868 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5591e86696e0 con 0x5591e8786ad0 2024-08-19T01:41:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.673+0000 7fe1537fe640 10 client.15855 handle_command_reply: tid=0 2024-08-19T01:41:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.673+0000 7fe17ce04740 1 client.15855 shutdown 2024-08-19T01:41:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.675+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fc2b0 msgr2=0x5591e8713460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.675+0000 7fe17ce04740 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fc2b0 0x5591e8713460 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009780 tx=0x7fe15c0092a0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.683+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 shutdown_connections 2024-08-19T01:41:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.683+0000 7fe17ce04740 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5591e8706b70 0x5591e870fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.684+0000 7fe17ce04740 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5591e86fc2b0 0x5591e8713460 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.684+0000 7fe17ce04740 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5591e8786ad0 0x5591e8788ef0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.684+0000 7fe17ce04740 1 --2- 172.21.15.64:0/9111868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5591e86fb990 0x5591e8712f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.684+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 >> 172.21.15.64:0/9111868 conn(0x5591e86f7f20 msgr2=0x5591e87041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.684+0000 7fe1537fe640 0 client.15855 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:41:53.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.684+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 shutdown_connections 2024-08-19T01:41:53.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.687+0000 7fe17ce04740 1 -- 172.21.15.64:0/9111868 wait complete. 2024-08-19T01:41:53.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.687+0000 7fe17ce04740 20 client.15855 trim_cache size 0 max 16384 2024-08-19T01:41:53.689 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 32, 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "5c5b6f92": { 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 32, 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:41:53.690 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 143.8, 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:41:53.691 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:41:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe1600a2cf0 msgr2=0x7fe1600a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe1600a2cf0 0x7fe1600a51b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe16c005e30 tx=0x7fe16c0095c0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174100b50 msgr2=0x7fe174198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:41:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174100b50 0x7fe174198170 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe16400d950 tx=0x7fe16400de10 comp rx=0 tx=0).stop 2024-08-19T01:41:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 shutdown_connections 2024-08-19T01:41:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe1600a2cf0 0x7fe1600a51b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe174102c20 0x7fe17419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1741022a0 0x7fe1741986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:41:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 --2- 172.21.15.64:0/1620900240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe174100b50 0x7fe174198170 secure :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe16400d950 tx=0x7fe16400de10 comp rx=0 tx=0).stop 2024-08-19T01:41:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.690+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 >> 172.21.15.64:0/1620900240 conn(0x7fe1740fc500 msgr2=0x7fe1740fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:41:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.691+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 shutdown_connections 2024-08-19T01:41:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:41:53.691+0000 7fe17aa06640 1 -- 172.21.15.64:0/1620900240 wait complete. 2024-08-19T01:41:53.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:53 smithi064 ceph-mon[18515]: pgmap v266: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-08-19T01:41:54.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:53 smithi123 ceph-mon[21829]: pgmap v266: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-08-19T01:41:54.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:53 smithi161 ceph-mon[21757]: pgmap v266: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-08-19T01:41:54.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 735337 8.02 MB/sec execute 326 sec latency 23749.343 ms 2024-08-19T01:41:54.442 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5c5b6f92', total seconds elapsed: 27.1 2024-08-19T01:41:55.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 736599 8.01 MB/sec execute 327 sec latency 211.535 ms 2024-08-19T01:41:56.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:55 smithi123 ceph-mon[21829]: pgmap v267: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-08-19T01:41:56.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:55 smithi161 ceph-mon[21757]: pgmap v267: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-08-19T01:41:56.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:55 smithi064 ceph-mon[18515]: pgmap v267: 129 pgs: 129 active+clean; 437 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-08-19T01:41:56.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 739062 8.03 MB/sec execute 328 sec latency 16.018 ms 2024-08-19T01:41:57.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 741583 8.04 MB/sec execute 329 sec latency 23.959 ms 2024-08-19T01:41:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:57 smithi123 ceph-mon[21829]: pgmap v268: 129 pgs: 129 active+clean; 440 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 825 KiB/s wr, 43 op/s 2024-08-19T01:41:58.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:57 smithi161 ceph-mon[21757]: pgmap v268: 129 pgs: 129 active+clean; 440 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 825 KiB/s wr, 43 op/s 2024-08-19T01:41:58.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:57 smithi064 ceph-mon[18515]: pgmap v268: 129 pgs: 129 active+clean; 440 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 825 KiB/s wr, 43 op/s 2024-08-19T01:41:58.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 743584 8.05 MB/sec execute 330 sec latency 11.857 ms 2024-08-19T01:41:59.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 746096 8.07 MB/sec execute 331 sec latency 15.158 ms 2024-08-19T01:42:00.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:41:59 smithi123 ceph-mon[21829]: pgmap v269: 129 pgs: 129 active+clean; 440 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 806 KiB/s wr, 40 op/s 2024-08-19T01:42:00.201 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:41:59 smithi161 ceph-mon[21757]: pgmap v269: 129 pgs: 129 active+clean; 440 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 806 KiB/s wr, 40 op/s 2024-08-19T01:42:00.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:41:59 smithi064 ceph-mon[18515]: pgmap v269: 129 pgs: 129 active+clean; 440 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 806 KiB/s wr, 40 op/s 2024-08-19T01:42:00.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 748443 8.08 MB/sec execute 332 sec latency 22.697 ms 2024-08-19T01:42:01.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:00 smithi123 ceph-mon[21829]: pgmap v270: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 115 op/s 2024-08-19T01:42:01.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:00 smithi161 ceph-mon[21757]: pgmap v270: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 115 op/s 2024-08-19T01:42:01.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:00 smithi064 ceph-mon[18515]: pgmap v270: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 115 op/s 2024-08-19T01:42:01.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 749203 8.08 MB/sec execute 333 sec latency 63.641 ms 2024-08-19T01:42:02.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 750914 8.07 MB/sec execute 334 sec latency 15.039 ms 2024-08-19T01:42:03.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 753387 8.09 MB/sec execute 335 sec latency 14.872 ms 2024-08-19T01:42:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:03 smithi123 ceph-mon[21829]: pgmap v271: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-08-19T01:42:03.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:03 smithi161 ceph-mon[21757]: pgmap v271: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-08-19T01:42:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:03 smithi064 ceph-mon[18515]: pgmap v271: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-08-19T01:42:04.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 755949 8.11 MB/sec execute 336 sec latency 20.474 ms 2024-08-19T01:42:05.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 757773 8.11 MB/sec execute 337 sec latency 16.358 ms 2024-08-19T01:42:05.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:05 smithi064 ceph-mon[18515]: pgmap v272: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-08-19T01:42:06.151 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:05 smithi161 ceph-mon[21757]: pgmap v272: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-08-19T01:42:06.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:05 smithi123 ceph-mon[21829]: pgmap v272: 129 pgs: 129 active+clean; 451 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 112 op/s 2024-08-19T01:42:06.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 760293 8.13 MB/sec execute 338 sec latency 14.095 ms 2024-08-19T01:42:07.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 762424 8.13 MB/sec execute 339 sec latency 17.178 ms 2024-08-19T01:42:07.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:07 smithi064 ceph-mon[18515]: pgmap v273: 129 pgs: 129 active+clean; 462 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 214 op/s 2024-08-19T01:42:07.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:07 smithi064 ceph-mon[18515]: osdmap e99: 12 total, 12 up, 12 in 2024-08-19T01:42:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:07 smithi123 ceph-mon[21829]: pgmap v273: 129 pgs: 129 active+clean; 462 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 214 op/s 2024-08-19T01:42:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:07 smithi123 ceph-mon[21829]: osdmap e99: 12 total, 12 up, 12 in 2024-08-19T01:42:08.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:07 smithi161 ceph-mon[21757]: pgmap v273: 129 pgs: 129 active+clean; 462 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 214 op/s 2024-08-19T01:42:08.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:07 smithi161 ceph-mon[21757]: osdmap e99: 12 total, 12 up, 12 in 2024-08-19T01:42:08.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 764596 8.15 MB/sec execute 340 sec latency 22.539 ms 2024-08-19T01:42:09.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 767275 8.17 MB/sec execute 341 sec latency 14.628 ms 2024-08-19T01:42:09.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:09 smithi064 ceph-mon[18515]: pgmap v275: 129 pgs: 129 active+clean; 462 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 211 op/s 2024-08-19T01:42:10.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:09 smithi123 ceph-mon[21829]: pgmap v275: 129 pgs: 129 active+clean; 462 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 211 op/s 2024-08-19T01:42:10.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:09 smithi161 ceph-mon[21757]: pgmap v275: 129 pgs: 129 active+clean; 462 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.5 MiB/s wr, 211 op/s 2024-08-19T01:42:10.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 769398 8.17 MB/sec execute 342 sec latency 17.059 ms 2024-08-19T01:42:11.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 771478 8.18 MB/sec execute 343 sec latency 11.854 ms 2024-08-19T01:42:11.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:11 smithi064 ceph-mon[18515]: pgmap v276: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-08-19T01:42:12.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:11 smithi123 ceph-mon[21829]: pgmap v276: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-08-19T01:42:12.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:11 smithi161 ceph-mon[21757]: pgmap v276: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-08-19T01:42:12.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 774203 8.21 MB/sec execute 344 sec latency 20.345 ms 2024-08-19T01:42:13.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 775554 8.20 MB/sec execute 345 sec latency 221.445 ms 2024-08-19T01:42:13.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:13 smithi064 ceph-mon[18515]: pgmap v277: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-08-19T01:42:13.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:13 smithi064 ceph-mon[18515]: osdmap e100: 12 total, 12 up, 12 in 2024-08-19T01:42:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:13 smithi123 ceph-mon[21829]: pgmap v277: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-08-19T01:42:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:13 smithi123 ceph-mon[21829]: osdmap e100: 12 total, 12 up, 12 in 2024-08-19T01:42:14.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:13 smithi161 ceph-mon[21757]: pgmap v277: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-08-19T01:42:14.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:13 smithi161 ceph-mon[21757]: osdmap e100: 12 total, 12 up, 12 in 2024-08-19T01:42:14.400 INFO:tasks.workunit.client.0.smithi064.stdout: 1 778074 8.22 MB/sec execute 346 sec latency 11.351 ms 2024-08-19T01:42:15.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 780613 8.23 MB/sec execute 347 sec latency 24.376 ms 2024-08-19T01:42:15.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:15 smithi064 ceph-mon[18515]: pgmap v279: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-08-19T01:42:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:15 smithi123 ceph-mon[21829]: pgmap v279: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-08-19T01:42:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:15 smithi161 ceph-mon[21757]: pgmap v279: 129 pgs: 129 active+clean; 474 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-08-19T01:42:16.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 782698 8.24 MB/sec execute 348 sec latency 23.618 ms 2024-08-19T01:42:17.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 785091 8.26 MB/sec execute 349 sec latency 13.393 ms 2024-08-19T01:42:17.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:17 smithi064 ceph-mon[18515]: pgmap v280: 129 pgs: 129 active+clean; 487 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-08-19T01:42:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:17 smithi123 ceph-mon[21829]: pgmap v280: 129 pgs: 129 active+clean; 487 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-08-19T01:42:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:17 smithi161 ceph-mon[21757]: pgmap v280: 129 pgs: 129 active+clean; 487 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-08-19T01:42:18.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 787083 8.26 MB/sec execute 350 sec latency 48.419 ms 2024-08-19T01:42:19.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 789104 8.27 MB/sec execute 351 sec latency 51.172 ms 2024-08-19T01:42:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:19 smithi123 ceph-mon[21829]: pgmap v281: 129 pgs: 129 active+clean; 487 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 186 op/s 2024-08-19T01:42:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:19 smithi161 ceph-mon[21757]: pgmap v281: 129 pgs: 129 active+clean; 487 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 186 op/s 2024-08-19T01:42:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:19 smithi064 ceph-mon[18515]: pgmap v281: 129 pgs: 129 active+clean; 487 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 186 op/s 2024-08-19T01:42:20.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 791781 8.29 MB/sec execute 352 sec latency 27.641 ms 2024-08-19T01:42:21.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 793467 8.29 MB/sec execute 353 sec latency 14.970 ms 2024-08-19T01:42:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:21 smithi123 ceph-mon[21829]: pgmap v282: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:42:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:21 smithi161 ceph-mon[21757]: pgmap v282: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:42:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:21 smithi064 ceph-mon[18515]: pgmap v282: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:42:22.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 796006 8.31 MB/sec execute 354 sec latency 14.354 ms 2024-08-19T01:42:23.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 798682 8.32 MB/sec execute 355 sec latency 20.893 ms 2024-08-19T01:42:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:23 smithi123 ceph-mon[21829]: pgmap v283: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:42:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:23 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:42:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:23 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:42:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:23 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:42:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:23 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:42:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:23 smithi161 ceph-mon[21757]: pgmap v283: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:42:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:23 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:42:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:23 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:42:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:23 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:42:24.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:23 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:42:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:23 smithi064 ceph-mon[18515]: pgmap v283: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.8 MiB/s wr, 178 op/s 2024-08-19T01:42:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:23 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:42:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:23 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:42:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:23 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:42:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:23 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:42:24.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 800381 8.33 MB/sec execute 356 sec latency 24.593 ms 2024-08-19T01:42:25.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 802749 8.34 MB/sec execute 357 sec latency 25.377 ms 2024-08-19T01:42:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:25 smithi123 ceph-mon[21829]: pgmap v284: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.7 MiB/s wr, 174 op/s 2024-08-19T01:42:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:25 smithi161 ceph-mon[21757]: pgmap v284: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.7 MiB/s wr, 174 op/s 2024-08-19T01:42:26.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:25 smithi064 ceph-mon[18515]: pgmap v284: 129 pgs: 129 active+clean; 499 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.7 MiB/s wr, 174 op/s 2024-08-19T01:42:26.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 804852 8.34 MB/sec execute 358 sec latency 13.486 ms 2024-08-19T01:42:27.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 807080 8.35 MB/sec execute 359 sec latency 12.095 ms 2024-08-19T01:42:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:27 smithi123 ceph-mon[21829]: pgmap v285: 129 pgs: 129 active+clean; 483 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 226 op/s 2024-08-19T01:42:28.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:27 smithi161 ceph-mon[21757]: pgmap v285: 129 pgs: 129 active+clean; 483 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 226 op/s 2024-08-19T01:42:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:27 smithi064 ceph-mon[18515]: pgmap v285: 129 pgs: 129 active+clean; 483 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 226 op/s 2024-08-19T01:42:28.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 809739 8.38 MB/sec execute 360 sec latency 16.988 ms 2024-08-19T01:42:29.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 811920 8.38 MB/sec execute 361 sec latency 12.505 ms 2024-08-19T01:42:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:29 smithi123 ceph-mon[21829]: pgmap v286: 129 pgs: 129 active+clean; 483 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-08-19T01:42:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:29 smithi161 ceph-mon[21757]: pgmap v286: 129 pgs: 129 active+clean; 483 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-08-19T01:42:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:29 smithi064 ceph-mon[18515]: pgmap v286: 129 pgs: 129 active+clean; 483 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-08-19T01:42:30.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 813650 8.39 MB/sec execute 362 sec latency 45.508 ms 2024-08-19T01:42:31.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 816061 8.40 MB/sec execute 363 sec latency 24.122 ms 2024-08-19T01:42:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:31 smithi123 ceph-mon[21829]: pgmap v287: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 245 op/s 2024-08-19T01:42:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:31 smithi161 ceph-mon[21757]: pgmap v287: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 245 op/s 2024-08-19T01:42:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:31 smithi064 ceph-mon[18515]: pgmap v287: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 245 op/s 2024-08-19T01:42:32.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 818049 8.41 MB/sec execute 364 sec latency 19.817 ms 2024-08-19T01:42:33.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 820527 8.42 MB/sec execute 365 sec latency 16.395 ms 2024-08-19T01:42:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:33 smithi123 ceph-mon[21829]: pgmap v288: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-08-19T01:42:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:33 smithi161 ceph-mon[21757]: pgmap v288: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-08-19T01:42:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:33 smithi064 ceph-mon[18515]: pgmap v288: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-08-19T01:42:34.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 822604 8.42 MB/sec execute 366 sec latency 45.983 ms 2024-08-19T01:42:35.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 824418 8.43 MB/sec execute 367 sec latency 84.107 ms 2024-08-19T01:42:36.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:35 smithi064 ceph-mon[18515]: pgmap v289: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-08-19T01:42:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:35 smithi123 ceph-mon[21829]: pgmap v289: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-08-19T01:42:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:35 smithi161 ceph-mon[21757]: pgmap v289: 129 pgs: 129 active+clean; 502 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-08-19T01:42:36.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 827300 8.45 MB/sec execute 368 sec latency 17.547 ms 2024-08-19T01:42:37.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 829207 8.45 MB/sec execute 369 sec latency 13.930 ms 2024-08-19T01:42:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:37 smithi123 ceph-mon[21829]: pgmap v290: 129 pgs: 129 active+clean; 512 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-08-19T01:42:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:37 smithi161 ceph-mon[21757]: pgmap v290: 129 pgs: 129 active+clean; 512 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-08-19T01:42:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:37 smithi064 ceph-mon[18515]: pgmap v290: 129 pgs: 129 active+clean; 512 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 268 op/s 2024-08-19T01:42:38.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 831508 8.46 MB/sec execute 370 sec latency 11.663 ms 2024-08-19T01:42:39.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 834163 8.48 MB/sec execute 371 sec latency 33.576 ms 2024-08-19T01:42:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:39 smithi123 ceph-mon[21829]: pgmap v291: 129 pgs: 129 active+clean; 512 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-08-19T01:42:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:39 smithi161 ceph-mon[21757]: pgmap v291: 129 pgs: 129 active+clean; 512 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-08-19T01:42:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:39 smithi064 ceph-mon[18515]: pgmap v291: 129 pgs: 129 active+clean; 512 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-08-19T01:42:40.401 INFO:tasks.workunit.client.0.smithi064.stdout: 1 835539 8.47 MB/sec execute 372 sec latency 47.028 ms 2024-08-19T01:42:41.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 838294 8.50 MB/sec execute 373 sec latency 14.796 ms 2024-08-19T01:42:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:41 smithi123 ceph-mon[21829]: pgmap v292: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.7 MiB/s wr, 292 op/s 2024-08-19T01:42:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:41 smithi161 ceph-mon[21757]: pgmap v292: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.7 MiB/s wr, 292 op/s 2024-08-19T01:42:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:41 smithi064 ceph-mon[18515]: pgmap v292: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.7 MiB/s wr, 292 op/s 2024-08-19T01:42:42.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 840466 8.50 MB/sec execute 374 sec latency 15.764 ms 2024-08-19T01:42:43.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 842605 8.51 MB/sec execute 375 sec latency 16.878 ms 2024-08-19T01:42:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:43 smithi123 ceph-mon[21829]: pgmap v293: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 198 op/s 2024-08-19T01:42:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:43 smithi161 ceph-mon[21757]: pgmap v293: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 198 op/s 2024-08-19T01:42:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:43 smithi064 ceph-mon[18515]: pgmap v293: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 198 op/s 2024-08-19T01:42:44.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 845290 8.53 MB/sec execute 376 sec latency 15.369 ms 2024-08-19T01:42:45.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 846881 8.52 MB/sec execute 377 sec latency 69.697 ms 2024-08-19T01:42:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:45 smithi123 ceph-mon[21829]: pgmap v294: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 198 op/s 2024-08-19T01:42:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:45 smithi161 ceph-mon[21757]: pgmap v294: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 198 op/s 2024-08-19T01:42:46.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:45 smithi064 ceph-mon[18515]: pgmap v294: 129 pgs: 129 active+clean; 525 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 198 op/s 2024-08-19T01:42:46.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 849245 8.54 MB/sec execute 378 sec latency 14.704 ms 2024-08-19T01:42:47.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 852053 8.56 MB/sec execute 379 sec latency 15.789 ms 2024-08-19T01:42:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:47 smithi123 ceph-mon[21829]: pgmap v295: 129 pgs: 129 active+clean; 532 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-08-19T01:42:48.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:47 smithi161 ceph-mon[21757]: pgmap v295: 129 pgs: 129 active+clean; 532 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-08-19T01:42:48.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:47 smithi064 ceph-mon[18515]: pgmap v295: 129 pgs: 129 active+clean; 532 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.3 MiB/s wr, 297 op/s 2024-08-19T01:42:48.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 853641 8.56 MB/sec execute 380 sec latency 36.172 ms 2024-08-19T01:42:49.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 856096 8.57 MB/sec execute 381 sec latency 39.638 ms 2024-08-19T01:42:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:49 smithi123 ceph-mon[21829]: pgmap v296: 129 pgs: 129 active+clean; 532 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 198 op/s 2024-08-19T01:42:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:49 smithi161 ceph-mon[21757]: pgmap v296: 129 pgs: 129 active+clean; 532 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 198 op/s 2024-08-19T01:42:50.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:49 smithi064 ceph-mon[18515]: pgmap v296: 129 pgs: 129 active+clean; 532 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 198 op/s 2024-08-19T01:42:50.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 858095 8.57 MB/sec execute 382 sec latency 72.606 ms 2024-08-19T01:42:51.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 860008 8.58 MB/sec execute 383 sec latency 23.716 ms 2024-08-19T01:42:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:51 smithi123 ceph-mon[21829]: pgmap v297: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.1 MiB/s wr, 293 op/s 2024-08-19T01:42:52.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:51 smithi161 ceph-mon[21757]: pgmap v297: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.1 MiB/s wr, 293 op/s 2024-08-19T01:42:52.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:51 smithi064 ceph-mon[18515]: pgmap v297: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.1 MiB/s wr, 293 op/s 2024-08-19T01:42:52.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 862856 8.60 MB/sec execute 384 sec latency 15.771 ms 2024-08-19T01:42:53.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 864448 8.59 MB/sec execute 385 sec latency 16.580 ms 2024-08-19T01:42:54.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:53 smithi123 ceph-mon[21829]: pgmap v298: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 192 op/s 2024-08-19T01:42:54.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:53 smithi161 ceph-mon[21757]: pgmap v298: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 192 op/s 2024-08-19T01:42:54.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:53 smithi064 ceph-mon[18515]: pgmap v298: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 192 op/s 2024-08-19T01:42:54.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 866976 8.61 MB/sec execute 386 sec latency 18.608 ms 2024-08-19T01:42:55.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 869192 8.62 MB/sec execute 387 sec latency 72.516 ms 2024-08-19T01:42:56.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:55 smithi123 ceph-mon[21829]: pgmap v299: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 192 op/s 2024-08-19T01:42:56.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:55 smithi161 ceph-mon[21757]: pgmap v299: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 192 op/s 2024-08-19T01:42:56.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:55 smithi064 ceph-mon[18515]: pgmap v299: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 192 op/s 2024-08-19T01:42:56.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 870990 8.62 MB/sec execute 388 sec latency 51.986 ms 2024-08-19T01:42:57.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 873609 8.64 MB/sec execute 389 sec latency 13.446 ms 2024-08-19T01:42:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:57 smithi123 ceph-mon[21829]: pgmap v300: 129 pgs: 129 active+clean; 503 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-08-19T01:42:58.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:57 smithi161 ceph-mon[21757]: pgmap v300: 129 pgs: 129 active+clean; 503 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-08-19T01:42:58.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:57 smithi064 ceph-mon[18515]: pgmap v300: 129 pgs: 129 active+clean; 503 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 296 op/s 2024-08-19T01:42:58.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 875470 8.63 MB/sec execute 390 sec latency 16.912 ms 2024-08-19T01:42:59.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 877669 8.64 MB/sec execute 391 sec latency 15.553 ms 2024-08-19T01:43:00.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:42:59 smithi123 ceph-mon[21829]: pgmap v301: 129 pgs: 129 active+clean; 503 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 198 op/s 2024-08-19T01:43:00.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:42:59 smithi161 ceph-mon[21757]: pgmap v301: 129 pgs: 129 active+clean; 503 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 198 op/s 2024-08-19T01:43:00.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:42:59 smithi064 ceph-mon[18515]: pgmap v301: 129 pgs: 129 active+clean; 503 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 198 op/s 2024-08-19T01:43:00.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 879804 8.65 MB/sec execute 392 sec latency 84.404 ms 2024-08-19T01:43:01.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 880220 8.64 MB/sec execute 393 sec latency 127.356 ms 2024-08-19T01:43:02.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:01 smithi123 ceph-mon[21829]: pgmap v302: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 300 op/s 2024-08-19T01:43:02.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:01 smithi161 ceph-mon[21757]: pgmap v302: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 300 op/s 2024-08-19T01:43:02.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:01 smithi064 ceph-mon[18515]: pgmap v302: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 300 op/s 2024-08-19T01:43:02.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 881584 8.63 MB/sec execute 394 sec latency 60.073 ms 2024-08-19T01:43:03.402 INFO:tasks.workunit.client.0.smithi064.stdout: 1 884234 8.65 MB/sec execute 395 sec latency 15.937 ms 2024-08-19T01:43:04.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:03 smithi123 ceph-mon[21829]: pgmap v303: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:43:04.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:03 smithi161 ceph-mon[21757]: pgmap v303: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:43:04.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:03 smithi064 ceph-mon[18515]: pgmap v303: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:43:04.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 885576 8.64 MB/sec execute 396 sec latency 223.425 ms 2024-08-19T01:43:05.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 887891 8.66 MB/sec execute 397 sec latency 21.856 ms 2024-08-19T01:43:06.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:05 smithi123 ceph-mon[21829]: pgmap v304: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:43:06.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:05 smithi161 ceph-mon[21757]: pgmap v304: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:43:06.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:05 smithi064 ceph-mon[18515]: pgmap v304: 129 pgs: 129 active+clean; 514 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:43:06.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 889225 8.65 MB/sec execute 398 sec latency 83.163 ms 2024-08-19T01:43:07.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 891249 8.66 MB/sec execute 399 sec latency 98.557 ms 2024-08-19T01:43:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:07 smithi123 ceph-mon[21829]: pgmap v305: 129 pgs: 129 active+clean; 523 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 332 op/s 2024-08-19T01:43:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:07 smithi123 ceph-mon[21829]: osdmap e101: 12 total, 12 up, 12 in 2024-08-19T01:43:08.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:07 smithi161 ceph-mon[21757]: pgmap v305: 129 pgs: 129 active+clean; 523 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 332 op/s 2024-08-19T01:43:08.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:07 smithi161 ceph-mon[21757]: osdmap e101: 12 total, 12 up, 12 in 2024-08-19T01:43:08.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:07 smithi064 ceph-mon[18515]: pgmap v305: 129 pgs: 129 active+clean; 523 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 332 op/s 2024-08-19T01:43:08.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:07 smithi064 ceph-mon[18515]: osdmap e101: 12 total, 12 up, 12 in 2024-08-19T01:43:08.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 892769 8.65 MB/sec execute 400 sec latency 22.484 ms 2024-08-19T01:43:09.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 894794 8.66 MB/sec execute 401 sec latency 40.153 ms 2024-08-19T01:43:09.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:09 smithi064 ceph-mon[18515]: pgmap v307: 129 pgs: 129 active+clean; 523 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.8 MiB/s wr, 274 op/s 2024-08-19T01:43:10.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:09 smithi123 ceph-mon[21829]: pgmap v307: 129 pgs: 129 active+clean; 523 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.8 MiB/s wr, 274 op/s 2024-08-19T01:43:10.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:09 smithi161 ceph-mon[21757]: pgmap v307: 129 pgs: 129 active+clean; 523 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.8 MiB/s wr, 274 op/s 2024-08-19T01:43:10.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 896518 8.66 MB/sec execute 402 sec latency 17.692 ms 2024-08-19T01:43:11.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 898351 8.67 MB/sec execute 403 sec latency 121.116 ms 2024-08-19T01:43:11.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:11 smithi064 ceph-mon[18515]: pgmap v308: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-08-19T01:43:12.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:11 smithi123 ceph-mon[21829]: pgmap v308: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-08-19T01:43:12.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:11 smithi161 ceph-mon[21757]: pgmap v308: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-08-19T01:43:12.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 899848 8.66 MB/sec execute 404 sec latency 149.713 ms 2024-08-19T01:43:13.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 901980 8.67 MB/sec execute 405 sec latency 41.650 ms 2024-08-19T01:43:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:13 smithi123 ceph-mon[21829]: pgmap v309: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-08-19T01:43:14.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:13 smithi161 ceph-mon[21757]: pgmap v309: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-08-19T01:43:14.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:13 smithi064 ceph-mon[18515]: pgmap v309: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 256 op/s 2024-08-19T01:43:14.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 903416 8.67 MB/sec execute 406 sec latency 63.260 ms 2024-08-19T01:43:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:14 smithi123 ceph-mon[21829]: osdmap e102: 12 total, 12 up, 12 in 2024-08-19T01:43:15.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:14 smithi161 ceph-mon[21757]: osdmap e102: 12 total, 12 up, 12 in 2024-08-19T01:43:15.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:14 smithi064 ceph-mon[18515]: osdmap e102: 12 total, 12 up, 12 in 2024-08-19T01:43:15.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 905605 8.68 MB/sec execute 407 sec latency 17.587 ms 2024-08-19T01:43:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:15 smithi123 ceph-mon[21829]: pgmap v311: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-08-19T01:43:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:15 smithi161 ceph-mon[21757]: pgmap v311: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-08-19T01:43:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:15 smithi064 ceph-mon[18515]: pgmap v311: 129 pgs: 129 active+clean; 533 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-08-19T01:43:16.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 906939 8.67 MB/sec execute 408 sec latency 46.392 ms 2024-08-19T01:43:17.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 908976 8.68 MB/sec execute 409 sec latency 148.774 ms 2024-08-19T01:43:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:17 smithi123 ceph-mon[21829]: pgmap v312: 129 pgs: 129 active+clean; 545 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 236 op/s 2024-08-19T01:43:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:17 smithi161 ceph-mon[21757]: pgmap v312: 129 pgs: 129 active+clean; 545 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 236 op/s 2024-08-19T01:43:18.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:17 smithi064 ceph-mon[18515]: pgmap v312: 129 pgs: 129 active+clean; 545 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 236 op/s 2024-08-19T01:43:18.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 910734 8.68 MB/sec execute 410 sec latency 15.838 ms 2024-08-19T01:43:19.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 912858 8.69 MB/sec execute 411 sec latency 35.726 ms 2024-08-19T01:43:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:19 smithi123 ceph-mon[21829]: pgmap v313: 129 pgs: 129 active+clean; 545 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 216 op/s 2024-08-19T01:43:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:19 smithi161 ceph-mon[21757]: pgmap v313: 129 pgs: 129 active+clean; 545 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 216 op/s 2024-08-19T01:43:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:19 smithi064 ceph-mon[18515]: pgmap v313: 129 pgs: 129 active+clean; 545 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 216 op/s 2024-08-19T01:43:20.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 914984 8.69 MB/sec execute 412 sec latency 26.599 ms 2024-08-19T01:43:21.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 916267 8.69 MB/sec execute 413 sec latency 134.943 ms 2024-08-19T01:43:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:21 smithi123 ceph-mon[21829]: pgmap v314: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 226 op/s 2024-08-19T01:43:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:21 smithi161 ceph-mon[21757]: pgmap v314: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 226 op/s 2024-08-19T01:43:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:21 smithi064 ceph-mon[18515]: pgmap v314: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 226 op/s 2024-08-19T01:43:22.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 917399 8.72 MB/sec execute 414 sec latency 60.043 ms 2024-08-19T01:43:22.658 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.0 2024-08-19T01:43:22.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:43:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:43:23.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 920221 8.74 MB/sec execute 415 sec latency 55.515 ms 2024-08-19T01:43:23.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 -- 172.21.15.64:0/3681327799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1078e0 msgr2=0x7fdaac107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:23.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3681327799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1078e0 0x7fdaac107cc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009890 tx=0x7fdaa802a380 comp rx=0 tx=0).stop 2024-08-19T01:43:23.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 -- 172.21.15.64:0/3681327799 shutdown_connections 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3681327799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac102280 0x7fdaac10e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3681327799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac1018e0 0x7fdaac101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3681327799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1078e0 0x7fdaac107cc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 -- 172.21.15.64:0/3681327799 >> 172.21.15.64:0/3681327799 conn(0x7fdaac0fd600 msgr2=0x7fdaac0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 -- 172.21.15.64:0/3681327799 shutdown_connections 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.533+0000 7fdab32c9640 1 -- 172.21.15.64:0/3681327799 wait complete. 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 Processor -- start 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 -- start start 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1018e0 0x7fdaac1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac102280 0x7fdaac198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac1078e0 0x7fdaac19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac111ce0 con 0x7fdaac1018e0 2024-08-19T01:43:23.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac111b60 con 0x7fdaac1078e0 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab32c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac111e60 con 0x7fdaac102280 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab1065640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1018e0 0x7fdaac1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab0864640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac102280 0x7fdaac198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab1065640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1018e0 0x7fdaac1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60204/0 (socket says 172.21.15.64:60204) 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab1065640 1 -- 172.21.15.64:0/3535845208 learned_addr learned my addr 172.21.15.64:0/3535845208 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.534+0000 7fdab1866640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac1078e0 0x7fdaac19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab1065640 1 -- 172.21.15.64:0/3535845208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac102280 msgr2=0x7fdaac198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab1065640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac102280 0x7fdaac198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab1065640 1 -- 172.21.15.64:0/3535845208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac1078e0 msgr2=0x7fdaac19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab1065640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac1078e0 0x7fdaac19ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab1065640 1 -- 172.21.15.64:0/3535845208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac19d170 con 0x7fdaac1018e0 2024-08-19T01:43:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab0864640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac102280 0x7fdaac198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:43:23.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab1065640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1018e0 0x7fdaac1981c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009860 tx=0x7fdaa8033d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:23.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdaa27fc640 1 -- 172.21.15.64:0/3535845208 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa803f020 con 0x7fdaac1018e0 2024-08-19T01:43:23.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdaa27fc640 1 -- 172.21.15.64:0/3535845208 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaa8032770 con 0x7fdaac1018e0 2024-08-19T01:43:23.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaac19d400 con 0x7fdaac1018e0 2024-08-19T01:43:23.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdaa27fc640 1 -- 172.21.15.64:0/3535845208 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8037040 con 0x7fdaac1018e0 2024-08-19T01:43:23.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.535+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaac19d850 con 0x7fdaac1018e0 2024-08-19T01:43:23.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.537+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaac101d40 con 0x7fdaac1018e0 2024-08-19T01:43:23.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.538+0000 7fdaa27fc640 1 -- 172.21.15.64:0/3535845208 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdaa8042020 con 0x7fdaac1018e0 2024-08-19T01:43:23.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.538+0000 7fdaa27fc640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda980a2cf0 0x7fda980a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:23.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.538+0000 7fdab0864640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda980a2cf0 0x7fda980a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:23.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.538+0000 7fdaa27fc640 1 -- 172.21.15.64:0/3535845208 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fdaa80ef780 con 0x7fdaac1018e0 2024-08-19T01:43:23.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.539+0000 7fdab0864640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda980a2cf0 0x7fda980a51b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdaac100d40 tx=0x7fda9c005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:23.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.543+0000 7fdaa27fc640 1 -- 172.21.15.64:0/3535845208 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fdaa8049020 con 0x7fdaac1018e0 2024-08-19T01:43:23.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.788+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdaac109eb0 con 0x7fdaac1018e0 2024-08-19T01:43:23.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.789+0000 7fdaa27fc640 1 -- 172.21.15.64:0/3535845208 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7fdaa80bc0a0 con 0x7fdaac1018e0 2024-08-19T01:43:23.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:43:23.790 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:43:23.791 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:43:23.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda980a2cf0 msgr2=0x7fda980a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:23.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda980a2cf0 0x7fda980a51b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdaac100d40 tx=0x7fda9c005f90 comp rx=0 tx=0).stop 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1018e0 msgr2=0x7fdaac1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1018e0 0x7fdaac1981c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009860 tx=0x7fdaa8033d10 comp rx=0 tx=0).stop 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 shutdown_connections 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fda980a2cf0 0x7fda980a51b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac1078e0 0x7fdaac19ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac102280 0x7fdaac198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 --2- 172.21.15.64:0/3535845208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac1018e0 0x7fdaac1981c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 >> 172.21.15.64:0/3535845208 conn(0x7fdaac0fd600 msgr2=0x7fdaac0feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 shutdown_connections 2024-08-19T01:43:23.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:23.791+0000 7fdab32c9640 1 -- 172.21.15.64:0/3535845208 wait complete. 2024-08-19T01:43:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:23 smithi123 ceph-mon[21829]: pgmap v315: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 226 op/s 2024-08-19T01:43:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:23 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:43:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:23 smithi161 ceph-mon[21757]: pgmap v315: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 226 op/s 2024-08-19T01:43:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:23 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:43:24.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:23 smithi064 ceph-mon[18515]: pgmap v315: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 226 op/s 2024-08-19T01:43:24.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:23 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:43:24.403 INFO:tasks.workunit.client.0.smithi064.stdout: 1 921953 8.74 MB/sec execute 416 sec latency 12.923 ms 2024-08-19T01:43:24.672 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 142.0 --await-for 45.0' 2024-08-19T01:43:24.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 142.0 --await-for 45.0 2024-08-19T01:43:24.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:24 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3535845208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:43:25.018 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:24 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3535845208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:43:25.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:24 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3535845208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:43:25.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 924190 8.75 MB/sec execute 417 sec latency 34.650 ms 2024-08-19T01:43:25.733 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:43:26.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: pgmap v316: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 220 op/s 2024-08-19T01:43:26.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.043 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:25 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: pgmap v316: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 220 op/s 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:25 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: pgmap v316: 129 pgs: 129 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 220 op/s 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:26.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:25 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe254817640 1 -- 172.21.15.64:0/1655904661 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2400319b0 con 0x7fe250101090 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe257a7d640 1 -- 172.21.15.64:0/1655904661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 msgr2=0x7fe25010dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe257a7d640 1 --2- 172.21.15.64:0/1655904661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25010dcc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe240009900 tx=0x7fe24002a410 comp rx=0 tx=0).stop 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe257a7d640 1 -- 172.21.15.64:0/1655904661 shutdown_connections 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe257a7d640 1 --2- 172.21.15.64:0/1655904661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe25010e200 0x7fe250110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe257a7d640 1 --2- 172.21.15.64:0/1655904661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25010dcc0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe257a7d640 1 --2- 172.21.15.64:0/1655904661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2501006e0 0x7fe250100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.318+0000 7fe257a7d640 1 -- 172.21.15.64:0/1655904661 >> 172.21.15.64:0/1655904661 conn(0x7fe2500fc500 msgr2=0x7fe2500fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.319+0000 7fe257a7d640 1 -- 172.21.15.64:0/1655904661 shutdown_connections 2024-08-19T01:43:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.319+0000 7fe257a7d640 1 -- 172.21.15.64:0/1655904661 wait complete. 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.319+0000 7fe257a7d640 1 Processor -- start 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.319+0000 7fe257a7d640 1 -- start start 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe257a7d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2501006e0 0x7fe25019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe257a7d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe257a7d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe25010e200 0x7fe25019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe257a7d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe250113fc0 con 0x7fe250101090 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe257a7d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe250113e40 con 0x7fe2501006e0 2024-08-19T01:43:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe257a7d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe250114140 con 0x7fe25010e200 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe255018640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe255018640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25019a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60218/0 (socket says 172.21.15.64:60218) 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe255018640 1 -- 172.21.15.64:0/2577331000 learned_addr learned my addr 172.21.15.64:0/2577331000 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe255819640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2501006e0 0x7fe25019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.320+0000 7fe25601a640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe25010e200 0x7fe25019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe255819640 1 -- 172.21.15.64:0/2577331000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe25010e200 msgr2=0x7fe25019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe255819640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe25010e200 0x7fe25019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe255819640 1 -- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 msgr2=0x7fe25019a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe255819640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25019a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe255819640 1 -- 172.21.15.64:0/2577331000 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25019f410 con 0x7fe2501006e0 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe255018640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe255819640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2501006e0 0x7fe25019a460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00e9f0 tx=0x7fe24c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe246ffd640 1 -- 172.21.15.64:0/2577331000 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c00cd80 con 0x7fe2501006e0 2024-08-19T01:43:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe246ffd640 1 -- 172.21.15.64:0/2577331000 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe24c0035f0 con 0x7fe2501006e0 2024-08-19T01:43:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe246ffd640 1 -- 172.21.15.64:0/2577331000 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c010690 con 0x7fe2501006e0 2024-08-19T01:43:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe25019f700 con 0x7fe2501006e0 2024-08-19T01:43:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.321+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2501a6fe0 con 0x7fe2501006e0 2024-08-19T01:43:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe246ffd640 1 -- 172.21.15.64:0/2577331000 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe24c003b30 con 0x7fe2501006e0 2024-08-19T01:43:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 Processor -- start 2024-08-19T01:43:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 -- start start 2024-08-19T01:43:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5158f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a515a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a515a2b20 0x561a515a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a514f4a70 con 0x561a5158f050 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561a5148c780 con 0x561a515a2b20 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.324+0000 7fe259e7b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561a514f0a30 con 0x561a5158f970 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe25601a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5158f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255018640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a515a2b20 0x561a515a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe25601a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5158f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60220/0 (socket says 172.21.15.64:60220) 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe25601a640 1 -- 172.21.15.64:0/343222864 learned_addr learned my addr 172.21.15.64:0/343222864 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255819640 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a515a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255018640 1 -- 172.21.15.64:0/343222864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 msgr2=0x561a515a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255018640 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a515a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255018640 1 -- 172.21.15.64:0/343222864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 msgr2=0x561a5158f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255018640 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5158f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255018640 1 -- 172.21.15.64:0/343222864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a51266b90 con 0x561a515a2b20 2024-08-19T01:43:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe25601a640 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5158f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:43:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe246ffd640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe23c0a2d70 0x7fe23c0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe246ffd640 1 -- 172.21.15.64:0/2577331000 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fe24c0cfba0 con 0x7fe2501006e0 2024-08-19T01:43:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe255018640 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a515a2b20 0x561a515a4f10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe240009230 tx=0x7fe240005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37260476e546eb44 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.325+0000 7fe25601a640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe23c0a2d70 0x7fe23c0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.326+0000 7fe23bfff640 1 -- 172.21.15.64:0/343222864 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240040070 con 0x561a515a2b20 2024-08-19T01:43:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.326+0000 7fe23bfff640 1 -- 172.21.15.64:0/343222864 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe240031970 con 0x561a515a2b20 2024-08-19T01:43:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.326+0000 7fe23bfff640 1 -- 172.21.15.64:0/343222864 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240035040 con 0x561a515a2b20 2024-08-19T01:43:26.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 -- 172.21.15.64:0/343222864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a515a2b20 msgr2=0x561a515a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a515a2b20 0x561a515a4f10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe240009230 tx=0x7fe240005dd0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe25601a640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe23c0a2d70 0x7fe23c0a5230 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248009500 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:26.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 -- 172.21.15.64:0/343222864 shutdown_connections 2024-08-19T01:43:26.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a515a2b20 0x561a515a4f10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a515a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 --2- 172.21.15.64:0/343222864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5158f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 -- 172.21.15.64:0/343222864 >> 172.21.15.64:0/343222864 conn(0x561a5158b6c0 msgr2=0x561a5158baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 -- 172.21.15.64:0/343222864 shutdown_connections 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 -- 172.21.15.64:0/343222864 wait complete. 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 -- start start 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.327+0000 7fe259e7b740 1 Processor -- start 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5159faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a5159ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a515a2b20 0x561a515a0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe259e7b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561a515a5680 con 0x561a5158f050 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe259e7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561a514af990 con 0x561a5158f970 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe259e7b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561a515a5800 con 0x561a515a2b20 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe259e7b740 10 client.0 _ll_register_callbacks cb 0x561a51204090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a5159ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe25601a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a515a2b20 0x561a515a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255819640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5159faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a5159ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:39124/0 (socket says 172.21.15.64:39124) 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe25601a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a515a2b20 0x561a515a0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:34026/0 (socket says 172.21.15.64:34026) 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 -- 172.21.15.64:0/4159620369 learned_addr learned my addr 172.21.15.64:0/4159620369 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 -- 172.21.15.64:0/4159620369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a515a2b20 msgr2=0x561a515a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a515a2b20 0x561a515a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 -- 172.21.15.64:0/4159620369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 msgr2=0x561a5159faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5159faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.328+0000 7fe255018640 1 -- 172.21.15.64:0/4159620369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a514ae6c0 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe255018640 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a5159ffe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2400055f0 tx=0x7fe240004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c64f2c39d6939e8 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe2397fa640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe2397fa640 1 -- 172.21.15.64:0/4159620369 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24004b070 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe2397fa640 1 -- 172.21.15.64:0/4159620369 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fe2400035f0 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe240031970 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe2397fa640 1 -- 172.21.15.64:0/4159620369 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe240057050 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe259e7b740 10 client.25331 fetch_fsmap learned FSMap version 29 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.329+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561a515b13f0 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe2397fa640 1 -- 172.21.15.64:0/4159620369 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe240031970 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe2397fa640 1 -- 172.21.15.64:0/4159620369 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe24003c7b0 con 0x561a5158f970 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe259e7b740 10 client.25331 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:43:26.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe259e7b740 10 client.25331 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe259e7b740 20 client.25331 populate_metadata read hostname 'smithi064' 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe259e7b740 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a515b2940 0x561a515b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe259e7b740 4 client.25331 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a515021a0 con 0x561a515b2940 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe255819640 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a515b2940 0x561a515b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe2397fa640 10 client.25331 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.330+0000 7fe255819640 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a515b2940 0x561a515b4d80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11cc22609a9d3c7 server_cookie=fc3508400a90f756 in_seq=0 out_seq=0 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe2397fa640 1 -- 172.21.15.64:0/4159620369 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561a515021a0 con 0x561a515b2940 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe2397fa640 10 client.25331 handle_command_reply: tid=0 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 client.25331 shutdown 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 msgr2=0x561a5159ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a5159ffe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2400055f0 tx=0x7fe240004270 comp rx=0 tx=0).stop 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 shutdown_connections 2024-08-19T01:43:26.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a515a2b20 0x561a515a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a515b2940 0x561a515b4d80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a5159ffe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 --2- 172.21.15.64:0/4159620369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a5158f050 0x561a5159faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 >> 172.21.15.64:0/4159620369 conn(0x561a5158b6c0 msgr2=0x561a51591350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.331+0000 7fe2397fa640 0 client.25331 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.332+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 shutdown_connections 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.332+0000 7fe259e7b740 1 -- 172.21.15.64:0/4159620369 wait complete. 2024-08-19T01:43:26.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.332+0000 7fe259e7b740 20 client.25331 trim_cache size 0 max 16384 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.350+0000 7fe259e7b740 1 Processor -- start 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.350+0000 7fe259e7b740 1 -- start start 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.350+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f050 0x561a5159afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a5159ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a5159cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe259e7b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe2501af630 con 0x561a515a2b20 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe259e7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2501acf00 con 0x561a5158f050 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe259e7b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe2501aad70 con 0x561a5158f970 2024-08-19T01:43:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f050 0x561a5159afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f050 0x561a5159afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:39126/0 (socket says 172.21.15.64:39126) 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 -- 172.21.15.64:0/128857910 learned_addr learned my addr 172.21.15.64:0/128857910 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255018640 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a5159ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe25601a640 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a5159cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 -- 172.21.15.64:0/128857910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 msgr2=0x561a5159ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a5159ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 -- 172.21.15.64:0/128857910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 msgr2=0x561a5159cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a5159cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 -- 172.21.15.64:0/128857910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a51589c70 con 0x561a5158f050 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe255819640 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f050 0x561a5159afd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe24c09b950 tx=0x7fe24c0d1f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ef1d0cb2fbedeb8 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.351+0000 7fe23b7fe640 1 -- 172.21.15.64:0/128857910 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c0d5070 con 0x561a5158f050 2024-08-19T01:43:26.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe23b7fe640 1 -- 172.21.15.64:0/128857910 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe24c09c350 con 0x561a5158f050 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe23b7fe640 1 -- 172.21.15.64:0/128857910 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c0037a0 con 0x561a5158f050 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 -- 172.21.15.64:0/128857910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f050 msgr2=0x561a5159afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f050 0x561a5159afd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe24c09b950 tx=0x7fe24c0d1f70 comp rx=0 tx=0).stop 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 -- 172.21.15.64:0/128857910 shutdown_connections 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a5159cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f970 0x561a5159ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 --2- 172.21.15.64:0/128857910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f050 0x561a5159afd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 -- 172.21.15.64:0/128857910 >> 172.21.15.64:0/128857910 conn(0x561a5158b6c0 msgr2=0x561a515a1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 -- 172.21.15.64:0/128857910 shutdown_connections 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.352+0000 7fe259e7b740 1 -- 172.21.15.64:0/128857910 wait complete. 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 -- start start 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 Processor -- start 2024-08-19T01:43:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f050 0x561a515baaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a515b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a515b8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe2501ab3c0 con 0x561a515a2b20 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2501ac830 con 0x561a5158f970 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe2501aba90 con 0x561a5158f050 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe259e7b740 10 client.0 _ll_register_callbacks cb 0x7fe248009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe25601a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a515b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe255819640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f050 0x561a515baaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe25601a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a515b8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60242/0 (socket says 172.21.15.64:60242) 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe25601a640 1 -- 172.21.15.64:0/3870797823 learned_addr learned my addr 172.21.15.64:0/3870797823 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.353+0000 7fe255819640 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f050 0x561a515baaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:34042/0 (socket says 172.21.15.64:34042) 2024-08-19T01:43:26.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe255018640 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a515b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe255819640 1 -- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 msgr2=0x561a515b7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe255819640 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a515b7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe255819640 1 -- 172.21.15.64:0/3870797823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 msgr2=0x561a515b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe255819640 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a515b8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe255819640 1 -- 172.21.15.64:0/3870797823 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a515b88a0 con 0x561a5158f050 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe255819640 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f050 0x561a515baaa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0a01c0 tx=0x7fe24c0cceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c22db63b880177 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe2397fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe2397fa640 1 -- 172.21.15.64:0/3870797823 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c0b1070 con 0x561a5158f050 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe24c09c350 con 0x561a5158f050 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe2397fa640 1 -- 172.21.15.64:0/3870797823 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fe24c0037a0 con 0x561a5158f050 2024-08-19T01:43:26.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe2397fa640 1 -- 172.21.15.64:0/3870797823 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe24c0d5070 con 0x561a5158f050 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe259e7b740 10 client.25318 fetch_fsmap learned FSMap version 29 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.354+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561a5161d320 con 0x561a5158f050 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe2397fa640 1 -- 172.21.15.64:0/3870797823 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe24c09c350 con 0x561a5158f050 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe2397fa640 1 -- 172.21.15.64:0/3870797823 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe24c09b0a0 con 0x561a5158f050 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe259e7b740 10 client.25318 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe259e7b740 10 client.25318 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe259e7b740 20 client.25318 populate_metadata read hostname 'smithi064' 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe259e7b740 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a5161dba0 0x561a5161ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe259e7b740 4 client.25318 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.0, "await_for": 45.0}] 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.0, "await_for": 45.0}) -- 0x561a514fc4f0 con 0x561a5161dba0 2024-08-19T01:43:26.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe255018640 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a5161dba0 0x561a5161ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:26.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.355+0000 7fe2397fa640 10 client.25318 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:26.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.356+0000 7fe255018640 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a5161dba0 0x561a5161ffc0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=752c6b3d99b938b6 server_cookie=62c2e380c6c23231 in_seq=0 out_seq=0 2024-08-19T01:43:26.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925321 8.74 MB/sec execute 418 sec latency 101.590 ms 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.456+0000 7fe2397fa640 1 -- 172.21.15.64:0/3870797823 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x561a514fc4f0 con 0x561a5161dba0 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.456+0000 7fe2397fa640 10 client.25318 handle_command_reply: tid=0 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.456+0000 7fe259e7b740 1 client.25318 shutdown 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f050 msgr2=0x561a515baaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f050 0x561a515baaa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0a01c0 tx=0x7fe24c0cceb0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 shutdown_connections 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561a515a2b20 0x561a515b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x561a5161dba0 0x561a5161ffc0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561a5158f970 0x561a515b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 --2- 172.21.15.64:0/3870797823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561a5158f050 0x561a515baaa0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe2397fa640 0 client.25318 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 >> 172.21.15.64:0/3870797823 conn(0x561a5158b6c0 msgr2=0x561a515a1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 shutdown_connections 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 1 -- 172.21.15.64:0/3870797823 wait complete. 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.457+0000 7fe259e7b740 20 client.25318 trim_cache size 0 max 16384 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:43:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 34, 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "ad7765f": { 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 34, 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 142.0, 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:43:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:26.461 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe23c0a2d70 msgr2=0x7fe23c0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe23c0a2d70 0x7fe23c0a5230 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248009500 comp rx=0 tx=0).stop 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2501006e0 msgr2=0x7fe25019a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2501006e0 0x7fe25019a460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00e9f0 tx=0x7fe24c00eeb0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 shutdown_connections 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe23c0a2d70 0x7fe23c0a5230 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe25010e200 0x7fe25019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250101090 0x7fe25019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 --2- 172.21.15.64:0/2577331000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2501006e0 0x7fe25019a460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 >> 172.21.15.64:0/2577331000 conn(0x7fe2500fc500 msgr2=0x7fe25010f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 shutdown_connections 2024-08-19T01:43:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:26.458+0000 7fe257a7d640 1 -- 172.21.15.64:0/2577331000 wait complete. 2024-08-19T01:43:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-08-19T01:43:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: Unable to set osd_memory_target on smithi123 to 184594636: error parsing value: Value '184594636' is below minimum 939524096 2024-08-19T01:43:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:26 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: Unable to set osd_memory_target on smithi123 to 184594636: error parsing value: Value '184594636' is below minimum 939524096 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:26 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-08-19T01:43:27.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: Unable to set osd_memory_target on smithi123 to 184594636: error parsing value: Value '184594636' is below minimum 939524096 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:26 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:27.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.72 MB/sec execute 419 sec latency 939.219 ms 2024-08-19T01:43:27.536 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ad7765f', quiesce took 4.9 seconds. Will release after: 17.1 2024-08-19T01:43:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:27 smithi123 ceph-mon[21829]: pgmap v317: 129 pgs: 129 active+clean; 523 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.9 MiB/s wr, 277 op/s 2024-08-19T01:43:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:27 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-08-19T01:43:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:27 smithi123 ceph-mon[21829]: Unable to set osd_memory_target on smithi161 to 184597504: error parsing value: Value '184597504' is below minimum 939524096 2024-08-19T01:43:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:27 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:27 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:27 smithi161 ceph-mon[21757]: pgmap v317: 129 pgs: 129 active+clean; 523 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.9 MiB/s wr, 277 op/s 2024-08-19T01:43:28.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:27 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-08-19T01:43:28.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:27 smithi161 ceph-mon[21757]: Unable to set osd_memory_target on smithi161 to 184597504: error parsing value: Value '184597504' is below minimum 939524096 2024-08-19T01:43:28.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:27 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:28.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:27 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:27 smithi064 ceph-mon[18515]: pgmap v317: 129 pgs: 129 active+clean; 523 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.9 MiB/s wr, 277 op/s 2024-08-19T01:43:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:27 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-08-19T01:43:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:27 smithi064 ceph-mon[18515]: Unable to set osd_memory_target on smithi161 to 184597504: error parsing value: Value '184597504' is below minimum 939524096 2024-08-19T01:43:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:27 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:28.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:27 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:28.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.70 MB/sec execute 420 sec latency 1939.320 ms 2024-08-19T01:43:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:43:29.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:28 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:43:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:28 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:43:29.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:28 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:43:29.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.68 MB/sec execute 421 sec latency 2939.415 ms 2024-08-19T01:43:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:29 smithi123 ceph-mon[21829]: pgmap v318: 129 pgs: 129 active+clean; 523 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-08-19T01:43:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:29 smithi123 ceph-mon[21829]: Adjusting osd_memory_target on smithi064 to 176.0M 2024-08-19T01:43:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:29 smithi123 ceph-mon[21829]: Unable to set osd_memory_target on smithi064 to 184603955: error parsing value: Value '184603955' is below minimum 939524096 2024-08-19T01:43:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:29 smithi161 ceph-mon[21757]: pgmap v318: 129 pgs: 129 active+clean; 523 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-08-19T01:43:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:29 smithi161 ceph-mon[21757]: Adjusting osd_memory_target on smithi064 to 176.0M 2024-08-19T01:43:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:29 smithi161 ceph-mon[21757]: Unable to set osd_memory_target on smithi064 to 184603955: error parsing value: Value '184603955' is below minimum 939524096 2024-08-19T01:43:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:29 smithi064 ceph-mon[18515]: pgmap v318: 129 pgs: 129 active+clean; 523 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 184 op/s 2024-08-19T01:43:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:29 smithi064 ceph-mon[18515]: Adjusting osd_memory_target on smithi064 to 176.0M 2024-08-19T01:43:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:29 smithi064 ceph-mon[18515]: Unable to set osd_memory_target on smithi064 to 184603955: error parsing value: Value '184603955' is below minimum 939524096 2024-08-19T01:43:30.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.66 MB/sec execute 422 sec latency 3939.522 ms 2024-08-19T01:43:31.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.64 MB/sec execute 423 sec latency 4939.662 ms 2024-08-19T01:43:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:31 smithi123 ceph-mon[21829]: pgmap v319: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 238 op/s 2024-08-19T01:43:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:31 smithi161 ceph-mon[21757]: pgmap v319: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 238 op/s 2024-08-19T01:43:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:31 smithi064 ceph-mon[18515]: pgmap v319: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 238 op/s 2024-08-19T01:43:32.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.62 MB/sec execute 424 sec latency 5939.747 ms 2024-08-19T01:43:33.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.60 MB/sec execute 425 sec latency 6939.850 ms 2024-08-19T01:43:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:33 smithi123 ceph-mon[21829]: pgmap v320: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-08-19T01:43:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:33 smithi161 ceph-mon[21757]: pgmap v320: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-08-19T01:43:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:33 smithi064 ceph-mon[18515]: pgmap v320: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-08-19T01:43:34.404 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.58 MB/sec execute 426 sec latency 7939.967 ms 2024-08-19T01:43:35.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.56 MB/sec execute 427 sec latency 8940.052 ms 2024-08-19T01:43:36.037 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:35 smithi064 ceph-mon[18515]: pgmap v321: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-08-19T01:43:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:35 smithi123 ceph-mon[21829]: pgmap v321: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-08-19T01:43:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:35 smithi161 ceph-mon[21757]: pgmap v321: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-08-19T01:43:36.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.54 MB/sec execute 428 sec latency 9940.108 ms 2024-08-19T01:43:37.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.52 MB/sec execute 429 sec latency 10940.149 ms 2024-08-19T01:43:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:37 smithi123 ceph-mon[21829]: pgmap v322: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-08-19T01:43:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:37 smithi161 ceph-mon[21757]: pgmap v322: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-08-19T01:43:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:37 smithi064 ceph-mon[18515]: pgmap v322: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 169 op/s 2024-08-19T01:43:38.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.50 MB/sec execute 430 sec latency 11940.227 ms 2024-08-19T01:43:39.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.48 MB/sec execute 431 sec latency 12940.311 ms 2024-08-19T01:43:39.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:39 smithi161 ceph-mon[21757]: pgmap v323: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-08-19T01:43:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:39 smithi123 ceph-mon[21829]: pgmap v323: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-08-19T01:43:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:39 smithi064 ceph-mon[18515]: pgmap v323: 129 pgs: 129 active+clean; 525 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-08-19T01:43:40.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.46 MB/sec execute 432 sec latency 13940.407 ms 2024-08-19T01:43:41.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.44 MB/sec execute 433 sec latency 14940.508 ms 2024-08-19T01:43:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:41 smithi161 ceph-mon[21757]: pgmap v324: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.3 MiB/s wr, 105 op/s 2024-08-19T01:43:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:41 smithi123 ceph-mon[21829]: pgmap v324: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.3 MiB/s wr, 105 op/s 2024-08-19T01:43:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:41 smithi064 ceph-mon[18515]: pgmap v324: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.3 MiB/s wr, 105 op/s 2024-08-19T01:43:42.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.42 MB/sec execute 434 sec latency 15940.615 ms 2024-08-19T01:43:43.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:42 smithi123 ceph-mon[21829]: pgmap v325: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-08-19T01:43:43.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:42 smithi161 ceph-mon[21757]: pgmap v325: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-08-19T01:43:43.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:42 smithi064 ceph-mon[18515]: pgmap v325: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-08-19T01:43:43.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.40 MB/sec execute 435 sec latency 16940.725 ms 2024-08-19T01:43:44.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.38 MB/sec execute 436 sec latency 17940.832 ms 2024-08-19T01:43:44.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:43:44.909 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:43:45.298 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:45 smithi064 ceph-mon[18515]: pgmap v326: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-08-19T01:43:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/583562326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 msgr2=0x7f9f7810b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/583562326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f7810b890 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009900 tx=0x7f9f7402a480 comp rx=0 tx=0).stop 2024-08-19T01:43:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/583562326 shutdown_connections 2024-08-19T01:43:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/583562326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f7810beb0 0x7f9f7810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/583562326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f7810b890 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/583562326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f78102a80 0x7f9f78102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/583562326 >> 172.21.15.64:0/583562326 conn(0x7f9f780fc500 msgr2=0x7f9f780fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/583562326 shutdown_connections 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.379+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/583562326 wait complete. 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 Processor -- start 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 -- start start 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f78102a80 0x7f9f78198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f78198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f7810beb0 0x7f9f7819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78111cb0 con 0x7f9f781033a0 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78111b30 con 0x7f9f7810beb0 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7f9ce640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78111e30 con 0x7f9f78102a80 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7cf69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f78198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7cf69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f78198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47446/0 (socket says 172.21.15.64:47446) 2024-08-19T01:43:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7cf69640 1 -- 172.21.15.64:0/3966192643 learned_addr learned my addr 172.21.15.64:0/3966192643 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7df6b640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f7810beb0 0x7f9f7819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.380+0000 7f9f7d76a640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f78102a80 0x7f9f78198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7cf69640 1 -- 172.21.15.64:0/3966192643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f78102a80 msgr2=0x7f9f78198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7cf69640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f78102a80 0x7f9f78198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7cf69640 1 -- 172.21.15.64:0/3966192643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f7810beb0 msgr2=0x7f9f7819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7cf69640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f7810beb0 0x7f9f7819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7cf69640 1 -- 172.21.15.64:0/3966192643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7819d0e0 con 0x7f9f781033a0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7cf69640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f78198670 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006010 tx=0x7f9f740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f6a7fc640 1 -- 172.21.15.64:0/3966192643 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7403f030 con 0x7f9f781033a0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f6a7fc640 1 -- 172.21.15.64:0/3966192643 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f74003ce0 con 0x7f9f781033a0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7819d370 con 0x7f9f781033a0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f6a7fc640 1 -- 172.21.15.64:0/3966192643 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74035070 con 0x7f9f781033a0 2024-08-19T01:43:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.381+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f781a4cb0 con 0x7f9f781033a0 2024-08-19T01:43:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.382+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f78102e60 con 0x7f9f781033a0 2024-08-19T01:43:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.383+0000 7f9f6a7fc640 1 -- 172.21.15.64:0/3966192643 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9f74042030 con 0x7f9f781033a0 2024-08-19T01:43:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.384+0000 7f9f6a7fc640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f9f640a2cf0 0x7f9f640a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.384+0000 7f9f7d76a640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f9f640a2cf0 0x7f9f640a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.384+0000 7f9f6a7fc640 1 -- 172.21.15.64:0/3966192643 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f9f740f00d0 con 0x7f9f781033a0 2024-08-19T01:43:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.385+0000 7f9f7d76a640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f9f640a2cf0 0x7f9f640a51b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c005fd0 tx=0x7f9f6c005850 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.387+0000 7f9f6a7fc640 1 -- 172.21.15.64:0/3966192643 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f9f740bc9a0 con 0x7f9f781033a0 2024-08-19T01:43:45.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.36 MB/sec execute 437 sec latency 18940.885 ms 2024-08-19T01:43:45.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:45 smithi123 ceph-mon[21829]: pgmap v326: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-08-19T01:43:45.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:45 smithi161 ceph-mon[21757]: pgmap v326: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-08-19T01:43:45.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.599+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f78106f50 con 0x7f9f781033a0 2024-08-19T01:43:45.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.600+0000 7f9f6a7fc640 1 -- 172.21.15.64:0/3966192643 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f9f740dae20 con 0x7f9f781033a0 2024-08-19T01:43:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:43:45.601 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:43:45.601 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:43:45.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f9f640a2cf0 msgr2=0x7f9f640a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:45.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f9f640a2cf0 0x7f9f640a51b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c005fd0 tx=0x7f9f6c005850 comp rx=0 tx=0).stop 2024-08-19T01:43:45.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 msgr2=0x7f9f78198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:45.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f78198670 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006010 tx=0x7f9f740042f0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 shutdown_connections 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f9f640a2cf0 0x7f9f640a51b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f7810beb0 0x7f9f7819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f781033a0 0x7f9f78198670 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 --2- 172.21.15.64:0/3966192643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f78102a80 0x7f9f78198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 >> 172.21.15.64:0/3966192643 conn(0x7f9f780fc500 msgr2=0x7f9f780fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 shutdown_connections 2024-08-19T01:43:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:45.602+0000 7f9f7f9ce640 1 -- 172.21.15.64:0/3966192643 wait complete. 2024-08-19T01:43:46.354 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:46 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3966192643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:43:46.386 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id ad7765f --release --await' 2024-08-19T01:43:46.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id ad7765f --release --await 2024-08-19T01:43:46.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925348 8.34 MB/sec execute 438 sec latency 19940.915 ms 2024-08-19T01:43:46.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:46 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3966192643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:43:46.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:46 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3966192643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:43:46.641 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:43:47.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 -- 172.21.15.64:0/1393992388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401007b0 msgr2=0x7f1c40100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 --2- 172.21.15.64:0/1393992388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401007b0 0x7f1c40100c10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c009930 tx=0x7f1c2c02a420 comp rx=0 tx=0).stop 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 -- 172.21.15.64:0/1393992388 shutdown_connections 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 --2- 172.21.15.64:0/1393992388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c40101150 0x7f1c4010d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 --2- 172.21.15.64:0/1393992388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401007b0 0x7f1c40100c10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 --2- 172.21.15.64:0/1393992388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c401067b0 0x7f1c40106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 -- 172.21.15.64:0/1393992388 >> 172.21.15.64:0/1393992388 conn(0x7f1c400fc4d0 msgr2=0x7f1c400fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 -- 172.21.15.64:0/1393992388 shutdown_connections 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 -- 172.21.15.64:0/1393992388 wait complete. 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 Processor -- start 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.135+0000 7f1c47158640 1 -- start start 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c47158640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c401007b0 0x7f1c40198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c47158640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c40101150 0x7f1c40199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c47158640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401067b0 0x7f1c401a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c47158640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1c40110bb0 con 0x7f1c401067b0 2024-08-19T01:43:47.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c47158640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c40110a30 con 0x7f1c401007b0 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c47158640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c40110d30 con 0x7f1c40101150 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401067b0 0x7f1c401a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401067b0 0x7f1c401a4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47460/0 (socket says 172.21.15.64:47460) 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 -- 172.21.15.64:0/2296972980 learned_addr learned my addr 172.21.15.64:0/2296972980 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c401007b0 0x7f1c40198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c40101150 0x7f1c40199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 -- 172.21.15.64:0/2296972980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c40101150 msgr2=0x7f1c40199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c40101150 0x7f1c40199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 -- 172.21.15.64:0/2296972980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c401007b0 msgr2=0x7f1c40198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c401007b0 0x7f1c40198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 -- 172.21.15.64:0/2296972980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c401a45a0 con 0x7f1c401067b0 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c40101150 0x7f1c40199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401067b0 0x7f1c401a4060 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1c3800eaa0 tx=0x7f1c3800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c3dffb640 1 -- 172.21.15.64:0/2296972980 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3800cdb0 con 0x7f1c401067b0 2024-08-19T01:43:47.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c3dffb640 1 -- 172.21.15.64:0/2296972980 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c38005020 con 0x7f1c401067b0 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.136+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c401a4770 con 0x7f1c401067b0 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.137+0000 7f1c3dffb640 1 -- 172.21.15.64:0/2296972980 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c380116b0 con 0x7f1c401067b0 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.137+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c401a4c80 con 0x7f1c401067b0 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.139+0000 7f1c3dffb640 1 -- 172.21.15.64:0/2296972980 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1c38005560 con 0x7f1c401067b0 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.139+0000 7f1c49556740 1 Processor -- start 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.139+0000 7f1c49556740 1 -- start start 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c3dffb640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1c300a2d70 0x7f1c300a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c49556740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f018425d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c49556740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f018430630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c49556740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f018432f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c49556740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55f01838bc60 con 0x55f018430b70 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c49556740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f018323b90 con 0x55f0184262b0 2024-08-19T01:43:47.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c3dffb640 1 -- 172.21.15.64:0/2296972980 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f1c380cfa30 con 0x7f1c401067b0 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c49556740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f018387c20 con 0x55f018425990 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1c300a2d70 0x7f1c300a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f018432f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f018432f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47472/0 (socket says 172.21.15.64:47472) 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c3ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f018430630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 -- 172.21.15.64:0/1922911100 learned_addr learned my addr 172.21.15.64:0/1922911100 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f018425d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 -- 172.21.15.64:0/1922911100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 msgr2=0x55f018425d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f018425d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 -- 172.21.15.64:0/1922911100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 msgr2=0x55f018430630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f018430630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1c300a2d70 0x7f1c300a5230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c000c00 tx=0x7f1c2c031710 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:47.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 -- 172.21.15.64:0/1922911100 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f01809b580 con 0x55f018430b70 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.140+0000 7f1c456f5640 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f018432f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1c38097fa0 tx=0x7f1c3809bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b12a2b81ca832ba server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c2affd640 1 -- 172.21.15.64:0/1922911100 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3801f070 con 0x55f018430b70 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c2affd640 1 -- 172.21.15.64:0/1922911100 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c380051d0 con 0x55f018430b70 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c2affd640 1 -- 172.21.15.64:0/1922911100 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3809b1f0 con 0x55f018430b70 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 -- 172.21.15.64:0/1922911100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 msgr2=0x55f018432f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f018432f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1c38097fa0 tx=0x7f1c3809bb40 comp rx=0 tx=0).stop 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 -- 172.21.15.64:0/1922911100 shutdown_connections 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f018432f60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f018430630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 --2- 172.21.15.64:0/1922911100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f018425d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 -- 172.21.15.64:0/1922911100 >> 172.21.15.64:0/1922911100 conn(0x55f018421f20 msgr2=0x55f018422300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 -- 172.21.15.64:0/1922911100 shutdown_connections 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.141+0000 7f1c49556740 1 -- 172.21.15.64:0/1922911100 wait complete. 2024-08-19T01:43:47.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 -- start start 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 Processor -- start 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f01842da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f01842dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55f018433740 con 0x55f018430b70 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f0183297c0 con 0x55f0184262b0 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f0184338c0 con 0x55f018425990 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c49556740 10 client.0 _ll_register_callbacks cb 0x55f018332f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47486/0 (socket says 172.21.15.64:47486) 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c456f5640 1 -- 172.21.15.64:0/2313815161 learned_addr learned my addr 172.21.15.64:0/2313815161 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.142+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f01842da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f01842dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c456f5640 1 -- 172.21.15.64:0/2313815161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 msgr2=0x55f01842da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f01842da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c456f5640 1 -- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 msgr2=0x55f01842dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f01842dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c456f5640 1 -- 172.21.15.64:0/2313815161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f018165960 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842e500 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009270 tx=0x7f1c38000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb2dced70c33062 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2313815161 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3801f040 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2313815161 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f1c3809b1a0 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c380051d0 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2313815161 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1c380ba270 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c49556740 10 client.15897 fetch_fsmap learned FSMap version 29 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f01843f7b0 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2313815161 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c380051d0 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.143+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2313815161 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f1c3800f040 con 0x55f018430b70 2024-08-19T01:43:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c49556740 10 client.15897 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c49556740 10 client.15897 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c49556740 20 client.15897 populate_metadata read hostname 'smithi064' 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c49556740 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f018440b00 0x55f018442ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c49556740 4 client.15897 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0183991d0 con 0x55f018440b00 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f018440b00 0x55f018442ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c1ffff640 10 client.15897 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.144+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f018440b00 0x55f018442ef0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4baaa8b0b3a868bb server_cookie=7553c0a8c5cdec72 in_seq=0 out_seq=0 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2313815161 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f0183991d0 con 0x55f018440b00 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c1ffff640 10 client.15897 handle_command_reply: tid=0 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 client.15897 shutdown 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 msgr2=0x55f01842e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842e500 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009270 tx=0x7f1c38000f40 comp rx=0 tx=0).stop 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 shutdown_connections 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842e500 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f018440b00 0x55f018442ef0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f0184262b0 0x55f01842dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 --2- 172.21.15.64:0/2313815161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018425990 0x55f01842da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 >> 172.21.15.64:0/2313815161 conn(0x55f018421f20 msgr2=0x55f018427c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.145+0000 7f1c1ffff640 0 client.15897 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.146+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 shutdown_connections 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.146+0000 7f1c49556740 1 -- 172.21.15.64:0/2313815161 wait complete. 2024-08-19T01:43:47.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.146+0000 7f1c49556740 20 client.15897 trim_cache size 0 max 16384 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 Processor -- start 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 -- start start 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f01842e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f0184262b0 0x55f018420f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401b1830 con 0x55f018430b70 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401af100 con 0x55f018425990 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c49556740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401acf70 con 0x55f0184262b0 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c456f5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47500/0 (socket says 172.21.15.64:47500) 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c456f5640 1 -- 172.21.15.64:0/2062318127 learned_addr learned my addr 172.21.15.64:0/2062318127 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f0184262b0 0x55f018420f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f01842e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c456f5640 1 -- 172.21.15.64:0/2062318127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f0184262b0 msgr2=0x55f018420f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f0184262b0 0x55f018420f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c456f5640 1 -- 172.21.15.64:0/2062318127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 msgr2=0x55f01842e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.163+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f01842e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c456f5640 1 -- 172.21.15.64:0/2062318127 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f018420470 con 0x55f018430b70 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f0184262b0 0x55f018420f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842ace0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1c38098510 tx=0x7f1c3809bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69c5282876c0dd9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c2a7fc640 1 -- 172.21.15.64:0/2062318127 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3801f030 con 0x55f018430b70 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c2a7fc640 1 -- 172.21.15.64:0/2062318127 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c380b9070 con 0x55f018430b70 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c2a7fc640 1 -- 172.21.15.64:0/2062318127 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c380052f0 con 0x55f018430b70 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c49556740 1 -- 172.21.15.64:0/2062318127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 msgr2=0x55f01842ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c49556740 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842ace0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1c38098510 tx=0x7f1c3809bb40 comp rx=0 tx=0).stop 2024-08-19T01:43:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c49556740 1 -- 172.21.15.64:0/2062318127 shutdown_connections 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.164+0000 7f1c49556740 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f018430b70 0x55f01842ace0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f0184262b0 0x55f018420f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 --2- 172.21.15.64:0/2062318127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f01842e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 -- 172.21.15.64:0/2062318127 >> 172.21.15.64:0/2062318127 conn(0x55f018421f20 msgr2=0x55f018420860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 -- 172.21.15.64:0/2062318127 shutdown_connections 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 -- 172.21.15.64:0/2062318127 wait complete. 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 -- start start 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 Processor -- start 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f018443820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f0184262b0 0x55f018443d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018430b70 0x55f0184442a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401ad5c0 con 0x55f0184262b0 2024-08-19T01:43:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401aea30 con 0x55f018425990 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401adc90 con 0x55f018430b70 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.165+0000 7f1c49556740 10 client.0 _ll_register_callbacks cb 0x7f1c3809c240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f0184262b0 0x55f018443d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f0184262b0 0x55f018443d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47502/0 (socket says 172.21.15.64:47502) 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 -- 172.21.15.64:0/2811661281 learned_addr learned my addr 172.21.15.64:0/2811661281 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f018443820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c456f5640 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018430b70 0x55f0184442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 -- 172.21.15.64:0/2811661281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018430b70 msgr2=0x55f0184442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018430b70 0x55f0184442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 -- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 msgr2=0x55f018443820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f018443820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 -- 172.21.15.64:0/2811661281 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0184aa440 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c3ffff640 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f0184262b0 0x55f018443d60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c005ab0 tx=0x7f1c2c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64dc020fbca836ae server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2811661281 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c035050 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2811661281 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f1c2c0046c0 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c380b9070 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2811661281 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1c2c05a050 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.166+0000 7f1c49556740 10 client.15903 fetch_fsmap learned FSMap version 29 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f0184ab6b0 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2811661281 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c380b9070 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2811661281 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f1c2c003990 con 0x55f0184262b0 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c49556740 10 client.15903 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c49556740 10 client.15903 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c49556740 20 client.15903 populate_metadata read hostname 'smithi064' 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c49556740 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f0184abe60 0x55f0184ae280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:43:47.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c49556740 4 client.15903 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ad7765f", "release": true, "await": true}] 2024-08-19T01:43:47.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ad7765f", "release": true, "await": true}) -- 0x55f0183936e0 con 0x55f0184abe60 2024-08-19T01:43:47.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.167+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f0184abe60 0x55f0184ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:43:47.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.168+0000 7f1c44ef4640 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f0184abe60 0x55f0184ae280 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc6fffad085cc87 server_cookie=1cb1380cfae939d7 in_seq=0 out_seq=0 2024-08-19T01:43:47.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.168+0000 7f1c1ffff640 10 client.15903 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c1ffff640 1 -- 172.21.15.64:0/2811661281 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55f0183936e0 con 0x55f0184abe60 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c1ffff640 10 client.15903 handle_command_reply: tid=0 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 client.15903 shutdown 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f0184262b0 msgr2=0x55f018443d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f0184262b0 0x55f018443d60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c005ab0 tx=0x7f1c2c02a420 comp rx=0 tx=0).stop 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 shutdown_connections 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f018430b70 0x55f0184442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f0184262b0 0x55f018443d60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55f0184abe60 0x55f0184ae280 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 --2- 172.21.15.64:0/2811661281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f018425990 0x55f018443820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.177+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 >> 172.21.15.64:0/2811661281 conn(0x55f018421f20 msgr2=0x55f01843c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c1ffff640 0 client.15903 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 shutdown_connections 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c49556740 1 -- 172.21.15.64:0/2811661281 wait complete. 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c49556740 20 client.15903 trim_cache size 0 max 16384 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:43:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 36, 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "ad7765f": { 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 36, 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 142.0, 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:43:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:43:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:43:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:43:47.182 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:43:47.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1c300a2d70 msgr2=0x7f1c300a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1c300a2d70 0x7f1c300a5230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c000c00 tx=0x7f1c2c031710 comp rx=0 tx=0).stop 2024-08-19T01:43:47.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401067b0 msgr2=0x7f1c401a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:43:47.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401067b0 0x7f1c401a4060 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1c3800eaa0 tx=0x7f1c3800ef60 comp rx=0 tx=0).stop 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 shutdown_connections 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1c300a2d70 0x7f1c300a5230 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c401067b0 0x7f1c401a4060 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c40101150 0x7f1c40199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 --2- 172.21.15.64:0/2296972980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c401007b0 0x7f1c40198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.178+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 >> 172.21.15.64:0/2296972980 conn(0x7f1c400fc4d0 msgr2=0x7f1c400fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.179+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 shutdown_connections 2024-08-19T01:43:47.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:43:47.179+0000 7f1c47158640 1 -- 172.21.15.64:0/2296972980 wait complete. 2024-08-19T01:43:47.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 925882 8.33 MB/sec execute 439 sec latency 20715.416 ms 2024-08-19T01:43:47.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:47 smithi123 ceph-mon[21829]: pgmap v327: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-08-19T01:43:47.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:47 smithi161 ceph-mon[21757]: pgmap v327: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-08-19T01:43:47.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:47 smithi064 ceph-mon[18515]: pgmap v327: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-08-19T01:43:48.011 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ad7765f', total seconds elapsed: 25.4 2024-08-19T01:43:48.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 928013 8.34 MB/sec execute 440 sec latency 15.653 ms 2024-08-19T01:43:49.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 930829 8.36 MB/sec execute 441 sec latency 15.156 ms 2024-08-19T01:43:49.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:49 smithi123 ceph-mon[21829]: pgmap v328: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-08-19T01:43:49.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:49 smithi161 ceph-mon[21757]: pgmap v328: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-08-19T01:43:49.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:49 smithi064 ceph-mon[18515]: pgmap v328: 129 pgs: 129 active+clean; 524 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-08-19T01:43:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 932523 8.35 MB/sec execute 442 sec latency 13.188 ms 2024-08-19T01:43:51.405 INFO:tasks.workunit.client.0.smithi064.stdout: 1 934815 8.37 MB/sec execute 443 sec latency 34.610 ms 2024-08-19T01:43:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:51 smithi123 ceph-mon[21829]: pgmap v329: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.5 MiB/s wr, 108 op/s 2024-08-19T01:43:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:51 smithi161 ceph-mon[21757]: pgmap v329: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.5 MiB/s wr, 108 op/s 2024-08-19T01:43:51.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:51 smithi064 ceph-mon[18515]: pgmap v329: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.5 MiB/s wr, 108 op/s 2024-08-19T01:43:52.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 936279 8.36 MB/sec execute 444 sec latency 53.461 ms 2024-08-19T01:43:53.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 938679 8.37 MB/sec execute 445 sec latency 13.442 ms 2024-08-19T01:43:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:53 smithi123 ceph-mon[21829]: pgmap v330: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-08-19T01:43:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:53 smithi161 ceph-mon[21757]: pgmap v330: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-08-19T01:43:53.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:53 smithi064 ceph-mon[18515]: pgmap v330: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-08-19T01:43:54.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 940503 8.37 MB/sec execute 446 sec latency 202.646 ms 2024-08-19T01:43:55.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 942555 8.38 MB/sec execute 447 sec latency 12.552 ms 2024-08-19T01:43:56.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:55 smithi123 ceph-mon[21829]: pgmap v331: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-08-19T01:43:56.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:55 smithi161 ceph-mon[21757]: pgmap v331: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-08-19T01:43:56.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:55 smithi064 ceph-mon[18515]: pgmap v331: 129 pgs: 129 active+clean; 530 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-08-19T01:43:56.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 945244 8.40 MB/sec execute 448 sec latency 13.741 ms 2024-08-19T01:43:57.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 946540 8.39 MB/sec execute 449 sec latency 48.326 ms 2024-08-19T01:43:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:57 smithi123 ceph-mon[21829]: pgmap v332: 129 pgs: 129 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-08-19T01:43:58.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:57 smithi161 ceph-mon[21757]: pgmap v332: 129 pgs: 129 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-08-19T01:43:58.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:57 smithi064 ceph-mon[18515]: pgmap v332: 129 pgs: 129 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-08-19T01:43:58.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 948796 8.40 MB/sec execute 450 sec latency 16.627 ms 2024-08-19T01:43:59.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 950344 8.40 MB/sec execute 451 sec latency 101.202 ms 2024-08-19T01:44:00.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:43:59 smithi123 ceph-mon[21829]: pgmap v333: 129 pgs: 129 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 141 op/s 2024-08-19T01:44:00.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:43:59 smithi161 ceph-mon[21757]: pgmap v333: 129 pgs: 129 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 141 op/s 2024-08-19T01:44:00.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:43:59 smithi064 ceph-mon[18515]: pgmap v333: 129 pgs: 129 active+clean; 541 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 141 op/s 2024-08-19T01:44:00.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 952467 8.41 MB/sec execute 452 sec latency 40.257 ms 2024-08-19T01:44:01.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 953225 8.40 MB/sec execute 453 sec latency 85.990 ms 2024-08-19T01:44:02.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:01 smithi123 ceph-mon[21829]: pgmap v334: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 231 op/s 2024-08-19T01:44:02.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:01 smithi161 ceph-mon[21757]: pgmap v334: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 231 op/s 2024-08-19T01:44:02.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:01 smithi064 ceph-mon[18515]: pgmap v334: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 231 op/s 2024-08-19T01:44:02.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 954797 8.39 MB/sec execute 454 sec latency 60.112 ms 2024-08-19T01:44:03.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 956533 8.40 MB/sec execute 455 sec latency 16.421 ms 2024-08-19T01:44:04.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:03 smithi123 ceph-mon[21829]: pgmap v335: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 176 op/s 2024-08-19T01:44:04.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:03 smithi161 ceph-mon[21757]: pgmap v335: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 176 op/s 2024-08-19T01:44:04.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:03 smithi064 ceph-mon[18515]: pgmap v335: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 176 op/s 2024-08-19T01:44:04.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 958187 8.40 MB/sec execute 456 sec latency 167.003 ms 2024-08-19T01:44:05.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 959883 8.40 MB/sec execute 457 sec latency 13.985 ms 2024-08-19T01:44:06.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:05 smithi064 ceph-mon[18515]: pgmap v336: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 176 op/s 2024-08-19T01:44:06.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:05 smithi123 ceph-mon[21829]: pgmap v336: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 176 op/s 2024-08-19T01:44:06.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:05 smithi161 ceph-mon[21757]: pgmap v336: 129 pgs: 129 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 176 op/s 2024-08-19T01:44:06.406 INFO:tasks.workunit.client.0.smithi064.stdout: 1 962283 8.41 MB/sec execute 458 sec latency 31.629 ms 2024-08-19T01:44:07.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 963641 8.41 MB/sec execute 459 sec latency 29.609 ms 2024-08-19T01:44:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:07 smithi123 ceph-mon[21829]: pgmap v337: 129 pgs: 129 active+clean; 582 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 6.2 MiB/s wr, 269 op/s 2024-08-19T01:44:08.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:07 smithi123 ceph-mon[21829]: osdmap e103: 12 total, 12 up, 12 in 2024-08-19T01:44:08.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:07 smithi161 ceph-mon[21757]: pgmap v337: 129 pgs: 129 active+clean; 582 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 6.2 MiB/s wr, 269 op/s 2024-08-19T01:44:08.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:07 smithi161 ceph-mon[21757]: osdmap e103: 12 total, 12 up, 12 in 2024-08-19T01:44:08.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:07 smithi064 ceph-mon[18515]: pgmap v337: 129 pgs: 129 active+clean; 582 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 6.2 MiB/s wr, 269 op/s 2024-08-19T01:44:08.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:07 smithi064 ceph-mon[18515]: osdmap e103: 12 total, 12 up, 12 in 2024-08-19T01:44:08.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 966341 8.43 MB/sec execute 460 sec latency 14.648 ms 2024-08-19T01:44:09.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 967497 8.42 MB/sec execute 461 sec latency 27.622 ms 2024-08-19T01:44:10.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:09 smithi123 ceph-mon[21829]: pgmap v339: 129 pgs: 129 active+clean; 582 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-08-19T01:44:10.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:09 smithi161 ceph-mon[21757]: pgmap v339: 129 pgs: 129 active+clean; 582 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-08-19T01:44:10.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:09 smithi064 ceph-mon[18515]: pgmap v339: 129 pgs: 129 active+clean; 582 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-08-19T01:44:10.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 970031 8.43 MB/sec execute 462 sec latency 16.099 ms 2024-08-19T01:44:11.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 971583 8.43 MB/sec execute 463 sec latency 14.340 ms 2024-08-19T01:44:12.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:11 smithi123 ceph-mon[21829]: pgmap v340: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-08-19T01:44:12.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:11 smithi161 ceph-mon[21757]: pgmap v340: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-08-19T01:44:12.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:11 smithi064 ceph-mon[18515]: pgmap v340: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-08-19T01:44:12.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 973895 8.44 MB/sec execute 464 sec latency 16.057 ms 2024-08-19T01:44:13.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 975875 8.44 MB/sec execute 465 sec latency 32.883 ms 2024-08-19T01:44:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:13 smithi123 ceph-mon[21829]: pgmap v341: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-08-19T01:44:14.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:13 smithi123 ceph-mon[21829]: osdmap e104: 12 total, 12 up, 12 in 2024-08-19T01:44:14.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:13 smithi161 ceph-mon[21757]: pgmap v341: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-08-19T01:44:14.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:13 smithi161 ceph-mon[21757]: osdmap e104: 12 total, 12 up, 12 in 2024-08-19T01:44:14.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:13 smithi064 ceph-mon[18515]: pgmap v341: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-08-19T01:44:14.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:13 smithi064 ceph-mon[18515]: osdmap e104: 12 total, 12 up, 12 in 2024-08-19T01:44:14.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 977610 8.45 MB/sec execute 466 sec latency 15.312 ms 2024-08-19T01:44:15.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 980211 8.46 MB/sec execute 467 sec latency 22.212 ms 2024-08-19T01:44:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:15 smithi123 ceph-mon[21829]: pgmap v343: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 104 op/s 2024-08-19T01:44:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:15 smithi161 ceph-mon[21757]: pgmap v343: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 104 op/s 2024-08-19T01:44:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:15 smithi064 ceph-mon[18515]: pgmap v343: 129 pgs: 129 active+clean; 575 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 104 op/s 2024-08-19T01:44:16.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 981550 8.45 MB/sec execute 468 sec latency 20.759 ms 2024-08-19T01:44:17.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 984088 8.47 MB/sec execute 469 sec latency 17.836 ms 2024-08-19T01:44:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:17 smithi123 ceph-mon[21829]: pgmap v344: 129 pgs: 129 active+clean; 591 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-08-19T01:44:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:17 smithi161 ceph-mon[21757]: pgmap v344: 129 pgs: 129 active+clean; 591 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-08-19T01:44:18.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:17 smithi064 ceph-mon[18515]: pgmap v344: 129 pgs: 129 active+clean; 591 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-08-19T01:44:18.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 985181 8.46 MB/sec execute 470 sec latency 38.297 ms 2024-08-19T01:44:19.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 987759 8.48 MB/sec execute 471 sec latency 13.104 ms 2024-08-19T01:44:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:19 smithi123 ceph-mon[21829]: pgmap v345: 129 pgs: 129 active+clean; 591 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.2 MiB/s wr, 173 op/s 2024-08-19T01:44:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:19 smithi161 ceph-mon[21757]: pgmap v345: 129 pgs: 129 active+clean; 591 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.2 MiB/s wr, 173 op/s 2024-08-19T01:44:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:19 smithi064 ceph-mon[18515]: pgmap v345: 129 pgs: 129 active+clean; 591 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.2 MiB/s wr, 173 op/s 2024-08-19T01:44:20.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 989430 8.47 MB/sec execute 472 sec latency 17.087 ms 2024-08-19T01:44:21.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 991631 8.48 MB/sec execute 473 sec latency 35.776 ms 2024-08-19T01:44:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:21 smithi123 ceph-mon[21829]: pgmap v346: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 167 op/s 2024-08-19T01:44:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:21 smithi161 ceph-mon[21757]: pgmap v346: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 167 op/s 2024-08-19T01:44:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:21 smithi064 ceph-mon[18515]: pgmap v346: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 167 op/s 2024-08-19T01:44:22.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 993560 8.48 MB/sec execute 474 sec latency 117.609 ms 2024-08-19T01:44:23.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 995162 8.49 MB/sec execute 475 sec latency 83.383 ms 2024-08-19T01:44:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:23 smithi123 ceph-mon[21829]: pgmap v347: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 167 op/s 2024-08-19T01:44:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:23 smithi161 ceph-mon[21757]: pgmap v347: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 167 op/s 2024-08-19T01:44:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:23 smithi064 ceph-mon[18515]: pgmap v347: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.1 MiB/s wr, 167 op/s 2024-08-19T01:44:24.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 997192 8.49 MB/sec execute 476 sec latency 25.427 ms 2024-08-19T01:44:25.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 998852 8.49 MB/sec execute 477 sec latency 15.092 ms 2024-08-19T01:44:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:25 smithi123 ceph-mon[21829]: pgmap v348: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 163 op/s 2024-08-19T01:44:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:25 smithi161 ceph-mon[21757]: pgmap v348: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 163 op/s 2024-08-19T01:44:26.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:25 smithi064 ceph-mon[18515]: pgmap v348: 129 pgs: 129 active+clean; 552 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 163 op/s 2024-08-19T01:44:26.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1001222 8.50 MB/sec execute 478 sec latency 17.890 ms 2024-08-19T01:44:27.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1002531 8.50 MB/sec execute 479 sec latency 60.871 ms 2024-08-19T01:44:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:27 smithi123 ceph-mon[21829]: pgmap v349: 129 pgs: 129 active+clean; 564 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 232 op/s 2024-08-19T01:44:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:27 smithi161 ceph-mon[21757]: pgmap v349: 129 pgs: 129 active+clean; 564 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 232 op/s 2024-08-19T01:44:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:27 smithi064 ceph-mon[18515]: pgmap v349: 129 pgs: 129 active+clean; 564 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 232 op/s 2024-08-19T01:44:28.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1004632 8.50 MB/sec execute 480 sec latency 23.282 ms 2024-08-19T01:44:28.869 INFO:tasks.quiescer.fs.[cephfs]:Total duration (40.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-19T01:44:28.869 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.15 2024-08-19T01:44:28.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:44:29.133 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:44:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:44:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:44:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:28 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:44:29.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:28 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:44:29.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:44:29.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:44:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:28 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:44:29.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:28 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:44:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:44:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:44:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:28 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:44:29.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:28 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:44:29.407 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1006081 8.50 MB/sec execute 481 sec latency 14.218 ms 2024-08-19T01:44:29.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 -- 172.21.15.64:0/750678929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c102250 msgr2=0x7f1b0c10e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 --2- 172.21.15.64:0/750678929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c102250 0x7f1b0c10e780 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009cd0 tx=0x7f1b0402a8e0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 -- 172.21.15.64:0/750678929 shutdown_connections 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 --2- 172.21.15.64:0/750678929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c102250 0x7f1b0c10e780 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 --2- 172.21.15.64:0/750678929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b0c1018b0 0x7f1b0c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 --2- 172.21.15.64:0/750678929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b0c1078b0 0x7f1b0c107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 -- 172.21.15.64:0/750678929 >> 172.21.15.64:0/750678929 conn(0x7f1b0c0fd5f0 msgr2=0x7f1b0c0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 -- 172.21.15.64:0/750678929 shutdown_connections 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 -- 172.21.15.64:0/750678929 wait complete. 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 Processor -- start 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.727+0000 7f1b1458d640 1 -- start start 2024-08-19T01:44:29.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b1458d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c1018b0 0x7f1b0c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b1458d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b0c102250 0x7f1b0c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b1458d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b0c1078b0 0x7f1b0c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b1458d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c111cb0 con 0x7f1b0c1018b0 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b1458d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c111b30 con 0x7f1b0c102250 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b1458d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c111e30 con 0x7f1b0c1078b0 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c1018b0 0x7f1b0c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c1018b0 0x7f1b0c198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35546/0 (socket says 172.21.15.64:35546) 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 -- 172.21.15.64:0/1123176334 learned_addr learned my addr 172.21.15.64:0/1123176334 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12b2a640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b0c1078b0 0x7f1b0c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b11b28640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b0c102250 0x7f1b0c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 -- 172.21.15.64:0/1123176334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b0c1078b0 msgr2=0x7f1b0c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b0c1078b0 0x7f1b0c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 -- 172.21.15.64:0/1123176334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b0c102250 msgr2=0x7f1b0c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:29.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b0c102250 0x7f1b0c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 -- 172.21.15.64:0/1123176334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0c19d120 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b12329640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c1018b0 0x7f1b0c198170 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1b0800e9f0 tx=0x7f1b0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.728+0000 7f1b037fe640 1 -- 172.21.15.64:0/1123176334 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0800cd80 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.729+0000 7f1b037fe640 1 -- 172.21.15.64:0/1123176334 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b08005500 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.729+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0c19d410 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.729+0000 7f1b037fe640 1 -- 172.21.15.64:0/1123176334 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b08010690 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.729+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b0c1a4cf0 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.730+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b0c101d10 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.732+0000 7f1b037fe640 1 -- 172.21.15.64:0/1123176334 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1b080046c0 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.732+0000 7f1b037fe640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1af80a2cf0 0x7f1af80a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.733+0000 7f1b037fe640 1 -- 172.21.15.64:0/1123176334 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f1b080cfde0 con 0x7f1b0c1018b0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.733+0000 7f1b11b28640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1af80a2cf0 0x7f1af80a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:29.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.733+0000 7f1b11b28640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1af80a2cf0 0x7f1af80a51b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0ffbe0 tx=0x7f1afc005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:29.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.735+0000 7f1b037fe640 1 -- 172.21.15.64:0/1123176334 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1b08053df0 con 0x7f1b0c1018b0 2024-08-19T01:44:29.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.966+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b0c1994d0 con 0x7f1b0c1018b0 2024-08-19T01:44:29.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.967+0000 7f1b037fe640 1 -- 172.21.15.64:0/1123176334 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f1b0809c5f0 con 0x7f1b0c1018b0 2024-08-19T01:44:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:44:29.969 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:44:29.970 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1af80a2cf0 msgr2=0x7f1af80a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1af80a2cf0 0x7f1af80a51b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0ffbe0 tx=0x7f1afc005f90 comp rx=0 tx=0).stop 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c1018b0 msgr2=0x7f1b0c198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c1018b0 0x7f1b0c198170 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1b0800e9f0 tx=0x7f1b0800edf0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 shutdown_connections 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1af80a2cf0 0x7f1af80a51b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b0c1078b0 0x7f1b0c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b0c102250 0x7f1b0c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 --2- 172.21.15.64:0/1123176334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b0c1018b0 0x7f1b0c198170 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 >> 172.21.15.64:0/1123176334 conn(0x7f1b0c0fd5f0 msgr2=0x7f1b0c0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:29.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 shutdown_connections 2024-08-19T01:44:29.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:29.969+0000 7f1b1458d640 1 -- 172.21.15.64:0/1123176334 wait complete. 2024-08-19T01:44:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:29 smithi123 ceph-mon[21829]: pgmap v350: 129 pgs: 129 active+clean; 564 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 157 op/s 2024-08-19T01:44:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:29 smithi161 ceph-mon[21757]: pgmap v350: 129 pgs: 129 active+clean; 564 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 157 op/s 2024-08-19T01:44:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:29 smithi064 ceph-mon[18515]: pgmap v350: 129 pgs: 129 active+clean; 564 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 157 op/s 2024-08-19T01:44:30.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1008872 8.52 MB/sec execute 482 sec latency 18.461 ms 2024-08-19T01:44:30.712 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0' 2024-08-19T01:44:30.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0 2024-08-19T01:44:30.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:30 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1123176334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:30.989 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:44:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:30 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1123176334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:31.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:30 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1123176334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:31.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010112 8.51 MB/sec execute 483 sec latency 38.105 ms 2024-08-19T01:44:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 -- 172.21.15.64:0/3601055212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1018b0 msgr2=0x7f0e3c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 --2- 172.21.15.64:0/3601055212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c101d10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009930 tx=0x7f0e2402a420 comp rx=0 tx=0).stop 2024-08-19T01:44:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 -- 172.21.15.64:0/3601055212 shutdown_connections 2024-08-19T01:44:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 --2- 172.21.15.64:0/3601055212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c102250 0x7f0e3c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 --2- 172.21.15.64:0/3601055212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c101d10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 --2- 172.21.15.64:0/3601055212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c1078b0 0x7f0e3c107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 -- 172.21.15.64:0/3601055212 >> 172.21.15.64:0/3601055212 conn(0x7f0e3c0fd5f0 msgr2=0x7f0e3c0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 -- 172.21.15.64:0/3601055212 shutdown_connections 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.518+0000 7f0e40bee640 1 -- 172.21.15.64:0/3601055212 wait complete. 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 Processor -- start 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 -- start start 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c102250 0x7f0e3c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1078b0 0x7f0e3c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c111cb0 con 0x7f0e3c1078b0 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c111b30 con 0x7f0e3c102250 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e40bee640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c111e30 con 0x7f0e3c1018b0 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e3ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1078b0 0x7f0e3c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e33fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c102250 0x7f0e3c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:58352/0 (socket says 172.21.15.64:58352) 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e3ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1078b0 0x7f0e3c19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35564/0 (socket says 172.21.15.64:35564) 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.519+0000 7f0e3a59c640 1 -- 172.21.15.64:0/2468125436 learned_addr learned my addr 172.21.15.64:0/2468125436 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e3ad9d640 1 -- 172.21.15.64:0/2468125436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c1018b0 msgr2=0x7f0e3c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e3ad9d640 1 -- 172.21.15.64:0/2468125436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c102250 msgr2=0x7f0e3c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c102250 0x7f0e3c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e3ad9d640 1 -- 172.21.15.64:0/2468125436 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3c19d1a0 con 0x7f0e3c1078b0 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e33fff640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c102250 0x7f0e3c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1078b0 0x7f0e3c19cac0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009d20 tx=0x7f0e34007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e337fe640 1 -- 172.21.15.64:0/2468125436 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34007ca0 con 0x7f0e3c1078b0 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e337fe640 1 -- 172.21.15.64:0/2468125436 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e34003e50 con 0x7f0e3c1078b0 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e337fe640 1 -- 172.21.15.64:0/2468125436 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3401b4b0 con 0x7f0e3c1078b0 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3c19d490 con 0x7f0e3c1078b0 2024-08-19T01:44:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.520+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3c074150 con 0x7f0e3c1078b0 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e337fe640 1 -- 172.21.15.64:0/2468125436 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0e34003970 con 0x7f0e3c1078b0 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 Processor -- start 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 -- start start 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd826a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd8cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56304bce7810 con 0x56304bd82be0 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56304bc8ec70 con 0x56304bd822c0 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e42fec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56304bce37d0 con 0x56304bd8d4a0 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd8cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd8cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35568/0 (socket says 172.21.15.64:35568) 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e3a59c640 1 -- 172.21.15.64:0/3810926972 learned_addr learned my addr 172.21.15.64:0/3810926972 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.523+0000 7f0e33fff640 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e33fff640 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e3a59c640 1 -- 172.21.15.64:0/3810926972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 msgr2=0x56304bd8f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e3a59c640 1 -- 172.21.15.64:0/3810926972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 msgr2=0x56304bd826a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd826a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e3a59c640 1 -- 172.21.15.64:0/3810926972 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304ba59ba0 con 0x56304bd82be0 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e33fff640 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e337fe640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0e280a2d70 0x7f0e280a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd8cf60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0115c0 tx=0x7f0e2c011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd26275bd32ed784 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e33fff640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0e280a2d70 0x7f0e280a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e30ff9640 1 -- 172.21.15.64:0/3810926972 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00f660 con 0x56304bd82be0 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e337fe640 1 -- 172.21.15.64:0/2468125436 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0e340d35d0 con 0x7f0e3c1078b0 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e30ff9640 1 -- 172.21.15.64:0/3810926972 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e2c00fce0 con 0x56304bd82be0 2024-08-19T01:44:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.524+0000 7f0e30ff9640 1 -- 172.21.15.64:0/3810926972 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c019de0 con 0x56304bd82be0 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 -- 172.21.15.64:0/3810926972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 msgr2=0x56304bd8cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd8cf60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0115c0 tx=0x7f0e2c011a80 comp rx=0 tx=0).stop 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e33fff640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0e280a2d70 0x7f0e280a5230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009900 tx=0x7f0e240058d0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 -- 172.21.15.64:0/3810926972 shutdown_connections 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd8cf60 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3810926972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd826a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 -- 172.21.15.64:0/3810926972 >> 172.21.15.64:0/3810926972 conn(0x56304bd7e850 msgr2=0x56304bd7ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 -- 172.21.15.64:0/3810926972 shutdown_connections 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.525+0000 7f0e42fec740 1 -- 172.21.15.64:0/3810926972 wait complete. 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 -- start start 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 Processor -- start 2024-08-19T01:44:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd8a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd82be0 0x56304bd8aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56304bd8ffb0 con 0x56304bd822c0 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56304bca29a0 con 0x56304bd82be0 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56304bd90130 con 0x56304bd8d4a0 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e42fec740 10 client.0 _ll_register_callbacks cb 0x56304bc90c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd8a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd8a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35576/0 (socket says 172.21.15.64:35576) 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e3a59c640 1 -- 172.21.15.64:0/1266640111 learned_addr learned my addr 172.21.15.64:0/1266640111 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e33fff640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd82be0 0x56304bd8aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.526+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e3a59c640 1 -- 172.21.15.64:0/1266640111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 msgr2=0x56304bd8afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e3a59c640 1 -- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd82be0 msgr2=0x56304bd8aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd82be0 0x56304bd8aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e3a59c640 1 -- 172.21.15.64:0/1266640111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56304bcae220 con 0x56304bd822c0 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e33fff640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd82be0 0x56304bd8aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd8a560 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c000c00 tx=0x7f0e2c019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f888b2cd4b06a8f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e21ffb640 1 -- 172.21.15.64:0/1266640111 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00fe90 con 0x56304bd822c0 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e21ffb640 1 -- 172.21.15.64:0/1266640111 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0e2c016050 con 0x56304bd822c0 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e2c00fce0 con 0x56304bd822c0 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e21ffb640 1 -- 172.21.15.64:0/1266640111 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0e2c02f050 con 0x56304bd822c0 2024-08-19T01:44:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e42fec740 10 client.15921 fetch_fsmap learned FSMap version 29 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.527+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56304bd9bd20 con 0x56304bd822c0 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e21ffb640 1 -- 172.21.15.64:0/1266640111 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e2c00fce0 con 0x56304bd822c0 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e21ffb640 1 -- 172.21.15.64:0/1266640111 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0e2c004750 con 0x56304bd822c0 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e42fec740 10 client.15921 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e42fec740 10 client.15921 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e42fec740 20 client.15921 populate_metadata read hostname 'smithi064' 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e42fec740 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304bd9d120 0x56304bd9f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e42fec740 4 client.15921 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56304bcf4f40 con 0x56304bd9d120 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e33fff640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304bd9d120 0x56304bd9f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e21ffb640 10 client.15921 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.528+0000 7f0e33fff640 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304bd9d120 0x56304bd9f510 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a13e28bfead47870 server_cookie=9b362b9395f7bf89 in_seq=0 out_seq=0 2024-08-19T01:44:31.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e21ffb640 1 -- 172.21.15.64:0/1266640111 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56304bcf4f40 con 0x56304bd9d120 2024-08-19T01:44:31.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e21ffb640 10 client.15921 handle_command_reply: tid=0 2024-08-19T01:44:31.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 client.15921 shutdown 2024-08-19T01:44:31.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 msgr2=0x56304bd8a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd8a560 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c000c00 tx=0x7f0e2c019780 comp rx=0 tx=0).stop 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 shutdown_connections 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd8afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304bd9d120 0x56304bd9f510 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd82be0 0x56304bd8aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 --2- 172.21.15.64:0/1266640111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd8a560 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e21ffb640 0 client.15921 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 >> 172.21.15.64:0/1266640111 conn(0x56304bd7e850 msgr2=0x56304bd7db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 shutdown_connections 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.529+0000 7f0e42fec740 1 -- 172.21.15.64:0/1266640111 wait complete. 2024-08-19T01:44:31.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.530+0000 7f0e42fec740 20 client.15921 trim_cache size 0 max 16384 2024-08-19T01:44:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 Processor -- start 2024-08-19T01:44:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 -- start start 2024-08-19T01:44:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd85ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd82be0 0x56304bd894a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd8d4a0 0x56304bda4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c1af660 con 0x56304bd822c0 2024-08-19T01:44:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c1acf30 con 0x56304bd8d4a0 2024-08-19T01:44:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e42fec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c1aada0 con 0x56304bd82be0 2024-08-19T01:44:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd85ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd8d4a0 0x56304bda4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd85ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35584/0 (socket says 172.21.15.64:35584) 2024-08-19T01:44:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3a59c640 1 -- 172.21.15.64:0/905598856 learned_addr learned my addr 172.21.15.64:0/905598856 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e33fff640 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd82be0 0x56304bd894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3ad9d640 1 -- 172.21.15.64:0/905598856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd82be0 msgr2=0x56304bd894a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd82be0 0x56304bd894a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3ad9d640 1 -- 172.21.15.64:0/905598856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 msgr2=0x56304bd85ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd85ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.548+0000 7f0e3ad9d640 1 -- 172.21.15.64:0/905598856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304bc8ec00 con 0x56304bd8d4a0 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.549+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd8d4a0 0x56304bda4c50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e340d0910 tx=0x7f0e3409b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93601cc4c88382a9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.549+0000 7f0e23fff640 1 -- 172.21.15.64:0/905598856 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e340b7030 con 0x56304bd8d4a0 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.549+0000 7f0e23fff640 1 -- 172.21.15.64:0/905598856 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e3409fde0 con 0x56304bd8d4a0 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.549+0000 7f0e23fff640 1 -- 172.21.15.64:0/905598856 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3409bd50 con 0x56304bd8d4a0 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 -- 172.21.15.64:0/905598856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd8d4a0 msgr2=0x56304bda4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd8d4a0 0x56304bda4c50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e340d0910 tx=0x7f0e3409b260 comp rx=0 tx=0).stop 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 -- 172.21.15.64:0/905598856 shutdown_connections 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd8d4a0 0x56304bda4c50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd82be0 0x56304bd894a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 --2- 172.21.15.64:0/905598856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd822c0 0x56304bd85ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 -- 172.21.15.64:0/905598856 >> 172.21.15.64:0/905598856 conn(0x56304bd7e850 msgr2=0x56304bd7db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 -- 172.21.15.64:0/905598856 shutdown_connections 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 -- 172.21.15.64:0/905598856 wait complete. 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 -- start start 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.550+0000 7f0e42fec740 1 Processor -- start 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd95b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd874f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e42fec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd87c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e42fec740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c1ab3f0 con 0x56304bd82be0 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e42fec740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c1ac860 con 0x56304bd822c0 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e42fec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c1abac0 con 0x56304bd8d4a0 2024-08-19T01:44:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e42fec740 10 client.0 _ll_register_callbacks cb 0x7f0e2c011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e33fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd874f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd95b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e3a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd95b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:41688/0 (socket says 172.21.15.64:41688) 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e3a59c640 1 -- 172.21.15.64:0/3985993486 learned_addr learned my addr 172.21.15.64:0/3985993486 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e3ad9d640 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd87c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e33fff640 1 -- 172.21.15.64:0/3985993486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 msgr2=0x56304bd87c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e33fff640 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd87c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e33fff640 1 -- 172.21.15.64:0/3985993486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 msgr2=0x56304bd95b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e33fff640 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd95b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.551+0000 7f0e33fff640 1 -- 172.21.15.64:0/3985993486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56304be06c20 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e33fff640 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd874f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005940 tx=0x7f0e24031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f8a96ec49a5f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e21ffb640 1 -- 172.21.15.64:0/3985993486 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e24043070 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e21ffb640 1 -- 172.21.15.64:0/3985993486 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0e24003c80 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e3409fde0 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e21ffb640 1 -- 172.21.15.64:0/3985993486 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0e24035040 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 10 client.15927 fetch_fsmap learned FSMap version 29 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56304be07e20 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e21ffb640 1 -- 172.21.15.64:0/3985993486 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e3409fde0 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e21ffb640 1 -- 172.21.15.64:0/3985993486 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0e2405b4c0 con 0x56304bd82be0 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 10 client.15927 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:44:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 10 client.15927 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:44:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 20 client.15927 populate_metadata read hostname 'smithi064' 2024-08-19T01:44:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304be08640 0x56304be0aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 4 client.15927 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}] 2024-08-19T01:44:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.552+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}) -- 0x56304bcef290 con 0x56304be08640 2024-08-19T01:44:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.553+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304be08640 0x56304be0aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.553+0000 7f0e3a59c640 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304be08640 0x56304be0aa60 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbc9a97eba0a6da6 server_cookie=a4c8972347554f55 in_seq=0 out_seq=0 2024-08-19T01:44:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.553+0000 7f0e21ffb640 10 client.15927 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:31.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e21ffb640 1 -- 172.21.15.64:0/3985993486 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56304bcef290 con 0x56304be08640 2024-08-19T01:44:31.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e21ffb640 10 client.15927 handle_command_reply: tid=0 2024-08-19T01:44:31.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 client.15927 shutdown 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 msgr2=0x56304bd874f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd874f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005940 tx=0x7f0e24031400 comp rx=0 tx=0).stop 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 shutdown_connections 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56304bd8d4a0 0x56304bd87c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56304bd82be0 0x56304bd874f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56304be08640 0x56304be0aa60 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 --2- 172.21.15.64:0/3985993486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56304bd822c0 0x56304bd95b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e21ffb640 0 client.15927 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 >> 172.21.15.64:0/3985993486 conn(0x56304bd7e850 msgr2=0x56304bd83ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 shutdown_connections 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 1 -- 172.21.15.64:0/3985993486 wait complete. 2024-08-19T01:44:31.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.614+0000 7f0e42fec740 20 client.15927 trim_cache size 0 max 16384 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 38, 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "2e2b1d45": { 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 38, 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:44:31.617 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 140.2, 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:44:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:44:31.619 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:44:31.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:31.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:31.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:31.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:31.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:31.619 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0e280a2d70 msgr2=0x7f0e280a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0e280a2d70 0x7f0e280a5230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009900 tx=0x7f0e240058d0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1078b0 msgr2=0x7f0e3c19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1078b0 0x7f0e3c19cac0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009d20 tx=0x7f0e34007590 comp rx=0 tx=0).stop 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 shutdown_connections 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0e280a2d70 0x7f0e280a5230 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e3c1078b0 0x7f0e3c19cac0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c102250 0x7f0e3c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 --2- 172.21.15.64:0/2468125436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e3c1018b0 0x7f0e3c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 >> 172.21.15.64:0/2468125436 conn(0x7f0e3c0fd5f0 msgr2=0x7f0e3c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:31.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 shutdown_connections 2024-08-19T01:44:31.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:31.615+0000 7f0e40bee640 1 -- 172.21.15.64:0/2468125436 wait complete. 2024-08-19T01:44:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:31 smithi123 ceph-mon[21829]: pgmap v351: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.1 MiB/s wr, 250 op/s 2024-08-19T01:44:32.183 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:31 smithi064 ceph-mon[18515]: pgmap v351: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.1 MiB/s wr, 250 op/s 2024-08-19T01:44:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:31 smithi161 ceph-mon[21757]: pgmap v351: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.1 MiB/s wr, 250 op/s 2024-08-19T01:44:32.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.50 MB/sec execute 484 sec latency 784.147 ms 2024-08-19T01:44:32.528 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2e2b1d45', quiesce took 3.7 seconds. Will release after: 16.45 2024-08-19T01:44:33.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.48 MB/sec execute 485 sec latency 1784.236 ms 2024-08-19T01:44:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:33 smithi123 ceph-mon[21829]: pgmap v352: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 185 op/s 2024-08-19T01:44:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:33 smithi161 ceph-mon[21757]: pgmap v352: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 185 op/s 2024-08-19T01:44:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:33 smithi064 ceph-mon[18515]: pgmap v352: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 185 op/s 2024-08-19T01:44:34.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.46 MB/sec execute 486 sec latency 2784.353 ms 2024-08-19T01:44:35.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.45 MB/sec execute 487 sec latency 3784.474 ms 2024-08-19T01:44:36.047 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:35 smithi064 ceph-mon[18515]: pgmap v353: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 185 op/s 2024-08-19T01:44:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:35 smithi123 ceph-mon[21829]: pgmap v353: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 185 op/s 2024-08-19T01:44:36.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:35 smithi161 ceph-mon[21757]: pgmap v353: 129 pgs: 129 active+clean; 577 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 185 op/s 2024-08-19T01:44:36.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.43 MB/sec execute 488 sec latency 4784.562 ms 2024-08-19T01:44:37.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.41 MB/sec execute 489 sec latency 5784.673 ms 2024-08-19T01:44:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:37 smithi123 ceph-mon[21829]: pgmap v354: 129 pgs: 129 active+clean; 581 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 233 op/s 2024-08-19T01:44:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:37 smithi161 ceph-mon[21757]: pgmap v354: 129 pgs: 129 active+clean; 581 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 233 op/s 2024-08-19T01:44:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:37 smithi064 ceph-mon[18515]: pgmap v354: 129 pgs: 129 active+clean; 581 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 233 op/s 2024-08-19T01:44:38.408 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.39 MB/sec execute 490 sec latency 6784.754 ms 2024-08-19T01:44:39.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.38 MB/sec execute 491 sec latency 7784.884 ms 2024-08-19T01:44:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:39 smithi123 ceph-mon[21829]: pgmap v355: 129 pgs: 129 active+clean; 581 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-08-19T01:44:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:39 smithi161 ceph-mon[21757]: pgmap v355: 129 pgs: 129 active+clean; 581 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-08-19T01:44:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:39 smithi064 ceph-mon[18515]: pgmap v355: 129 pgs: 129 active+clean; 581 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-08-19T01:44:40.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.36 MB/sec execute 492 sec latency 8784.983 ms 2024-08-19T01:44:41.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:40 smithi123 ceph-mon[21829]: pgmap v356: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 161 op/s 2024-08-19T01:44:41.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:40 smithi161 ceph-mon[21757]: pgmap v356: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 161 op/s 2024-08-19T01:44:41.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:40 smithi064 ceph-mon[18515]: pgmap v356: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 161 op/s 2024-08-19T01:44:41.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.34 MB/sec execute 493 sec latency 9785.095 ms 2024-08-19T01:44:42.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.33 MB/sec execute 494 sec latency 10785.192 ms 2024-08-19T01:44:43.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.31 MB/sec execute 495 sec latency 11785.291 ms 2024-08-19T01:44:43.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:43 smithi123 ceph-mon[21829]: pgmap v357: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-08-19T01:44:43.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:43 smithi161 ceph-mon[21757]: pgmap v357: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-08-19T01:44:43.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:43 smithi064 ceph-mon[18515]: pgmap v357: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-08-19T01:44:44.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.29 MB/sec execute 496 sec latency 12785.409 ms 2024-08-19T01:44:45.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.28 MB/sec execute 497 sec latency 13785.513 ms 2024-08-19T01:44:45.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:45 smithi123 ceph-mon[21829]: pgmap v358: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-08-19T01:44:45.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:45 smithi161 ceph-mon[21757]: pgmap v358: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-08-19T01:44:45.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:45 smithi064 ceph-mon[18515]: pgmap v358: 129 pgs: 129 active+clean; 583 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-08-19T01:44:46.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.26 MB/sec execute 498 sec latency 14785.624 ms 2024-08-19T01:44:47.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.24 MB/sec execute 499 sec latency 15785.716 ms 2024-08-19T01:44:47.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:47 smithi123 ceph-mon[21829]: pgmap v359: 129 pgs: 129 active+clean; 589 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-08-19T01:44:47.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:47 smithi161 ceph-mon[21757]: pgmap v359: 129 pgs: 129 active+clean; 589 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-08-19T01:44:47.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:47 smithi064 ceph-mon[18515]: pgmap v359: 129 pgs: 129 active+clean; 589 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 2.3 MiB/s wr, 88 op/s 2024-08-19T01:44:48.409 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.23 MB/sec execute 500 sec latency 16785.753 ms 2024-08-19T01:44:48.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:44:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:44:49.362 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:49 smithi064 ceph-mon[18515]: pgmap v360: 129 pgs: 129 active+clean; 589 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:44:49.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.21 MB/sec execute 501 sec latency 17785.852 ms 2024-08-19T01:44:49.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:49 smithi123 ceph-mon[21829]: pgmap v360: 129 pgs: 129 active+clean; 589 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:44:49.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:49 smithi161 ceph-mon[21757]: pgmap v360: 129 pgs: 129 active+clean; 589 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:44:49.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.753+0000 7f2ffc686640 1 -- 172.21.15.64:0/938156372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff41078b0 msgr2=0x7f2ff4107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:49.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.753+0000 7f2ffc686640 1 --2- 172.21.15.64:0/938156372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff41078b0 0x7f2ff4107c90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009930 tx=0x7f2ff002a420 comp rx=0 tx=0).stop 2024-08-19T01:44:49.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- 172.21.15.64:0/938156372 shutdown_connections 2024-08-19T01:44:49.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 --2- 172.21.15.64:0/938156372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ff4102250 0x7f2ff410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 --2- 172.21.15.64:0/938156372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ff41018b0 0x7f2ff4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 --2- 172.21.15.64:0/938156372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff41078b0 0x7f2ff4107c90 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- 172.21.15.64:0/938156372 >> 172.21.15.64:0/938156372 conn(0x7f2ff40fd5f0 msgr2=0x7f2ff40ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- 172.21.15.64:0/938156372 shutdown_connections 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- 172.21.15.64:0/938156372 wait complete. 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 Processor -- start 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- start start 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ff41018b0 0x7f2ff4074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff4102250 0x7f2ff4074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ff41078b0 0x7f2ff4074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4111cb0 con 0x7f2ff4102250 2024-08-19T01:44:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4111b30 con 0x7f2ff41078b0 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.754+0000 7f2ffc686640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4111e30 con 0x7f2ff41018b0 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff4102250 0x7f2ff4074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff4102250 0x7f2ff4074760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54464/0 (socket says 172.21.15.64:54464) 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 -- 172.21.15.64:0/2053760684 learned_addr learned my addr 172.21.15.64:0/2053760684 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ffa422640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ff41018b0 0x7f2ff4074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ffac23640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ff41078b0 0x7f2ff4074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 -- 172.21.15.64:0/2053760684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ff41018b0 msgr2=0x7f2ff4074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ff41018b0 0x7f2ff4074220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 -- 172.21.15.64:0/2053760684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ff41078b0 msgr2=0x7f2ff4074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ff41078b0 0x7f2ff4074ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 -- 172.21.15.64:0/2053760684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff41a66c0 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ff9c21640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff4102250 0x7f2ff4074760 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c970 tx=0x7f2fe400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2feb7fe640 1 -- 172.21.15.64:0/2053760684 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4007bb0 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff41a69b0 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2feb7fe640 1 -- 172.21.15.64:0/2053760684 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fe4007d50 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2feb7fe640 1 -- 172.21.15.64:0/2053760684 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400f450 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.755+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff41a6ea0 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.756+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff4101d10 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.757+0000 7f2feb7fe640 1 -- 172.21.15.64:0/2053760684 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2fe4005000 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.758+0000 7f2feb7fe640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2fe00a2cf0 0x7f2fe00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.758+0000 7f2ffa422640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2fe00a2cf0 0x7f2fe00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.758+0000 7f2feb7fe640 1 -- 172.21.15.64:0/2053760684 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f2fe40cf680 con 0x7f2ff4102250 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.759+0000 7f2ffa422640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2fe00a2cf0 0x7f2fe00a51b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009900 tx=0x7f2ff00058d0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:49.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.762+0000 7f2feb7fe640 1 -- 172.21.15.64:0/2053760684 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f2fe4014030 con 0x7f2ff4102250 2024-08-19T01:44:49.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.973+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ff4109e20 con 0x7f2ff4102250 2024-08-19T01:44:49.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.974+0000 7f2feb7fe640 1 -- 172.21.15.64:0/2053760684 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f2fe409be90 con 0x7f2ff4102250 2024-08-19T01:44:49.976 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:44:49.976 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:44:49.976 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:44:49.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2fe00a2cf0 msgr2=0x7f2fe00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2fe00a2cf0 0x7f2fe00a51b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009900 tx=0x7f2ff00058d0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff4102250 msgr2=0x7f2ff4074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff4102250 0x7f2ff4074760 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c970 tx=0x7f2fe400ce30 comp rx=0 tx=0).stop 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 shutdown_connections 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2fe00a2cf0 0x7f2fe00a51b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ff41078b0 0x7f2ff4074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff4102250 0x7f2ff4074760 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 --2- 172.21.15.64:0/2053760684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ff41018b0 0x7f2ff4074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.976+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 >> 172.21.15.64:0/2053760684 conn(0x7f2ff40fd5f0 msgr2=0x7f2ff40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:49.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.977+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 shutdown_connections 2024-08-19T01:44:49.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:49.977+0000 7f2ffc686640 1 -- 172.21.15.64:0/2053760684 wait complete. 2024-08-19T01:44:50.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:50 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2053760684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:50.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.19 MB/sec execute 502 sec latency 18785.936 ms 2024-08-19T01:44:50.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:50 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2053760684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:50.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:50 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2053760684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:50.698 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 2e2b1d45 --release --await' 2024-08-19T01:44:50.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 2e2b1d45 --release --await 2024-08-19T01:44:50.956 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:44:51.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1010466 8.18 MB/sec execute 503 sec latency 19785.992 ms 2024-08-19T01:44:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:51 smithi123 ceph-mon[21829]: pgmap v361: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-08-19T01:44:51.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:51 smithi064 ceph-mon[18515]: pgmap v361: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-08-19T01:44:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:51 smithi161 ceph-mon[21757]: pgmap v361: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.516+0000 7f04252da640 1 -- 172.21.15.64:0/3315831952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201007b0 msgr2=0x7f0420100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.516+0000 7f04252da640 1 --2- 172.21.15.64:0/3315831952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201007b0 0x7f0420100c10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0408009930 tx=0x7f040802a420 comp rx=0 tx=0).stop 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.516+0000 7f04252da640 1 -- 172.21.15.64:0/3315831952 shutdown_connections 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.516+0000 7f04252da640 1 --2- 172.21.15.64:0/3315831952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0420101150 0x7f042010d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.516+0000 7f04252da640 1 --2- 172.21.15.64:0/3315831952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201007b0 0x7f0420100c10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.516+0000 7f04252da640 1 --2- 172.21.15.64:0/3315831952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04201067b0 0x7f0420106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.516+0000 7f04252da640 1 -- 172.21.15.64:0/3315831952 >> 172.21.15.64:0/3315831952 conn(0x7f04200fc4d0 msgr2=0x7f04200fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 -- 172.21.15.64:0/3315831952 shutdown_connections 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 -- 172.21.15.64:0/3315831952 wait complete. 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 Processor -- start 2024-08-19T01:44:51.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 -- start start 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04201007b0 0x7f04201981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0420101150 0x7f04201986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201067b0 0x7f042019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0420110bb0 con 0x7f04201067b0 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0420110a30 con 0x7f0420101150 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f04252da640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0420110d30 con 0x7f04201007b0 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f041f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201067b0 0x7f042019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f041f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201067b0 0x7f042019ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54474/0 (socket says 172.21.15.64:54474) 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f041f7fe640 1 -- 172.21.15.64:0/3034662117 learned_addr learned my addr 172.21.15.64:0/3034662117 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.517+0000 7f0417fff640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0420101150 0x7f04201986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041effd640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04201007b0 0x7f04201981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041f7fe640 1 -- 172.21.15.64:0/3034662117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04201007b0 msgr2=0x7f04201981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041f7fe640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04201007b0 0x7f04201981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041f7fe640 1 -- 172.21.15.64:0/3034662117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0420101150 msgr2=0x7f04201986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041f7fe640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0420101150 0x7f04201986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041f7fe640 1 -- 172.21.15.64:0/3034662117 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042019d150 con 0x7f04201067b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041f7fe640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201067b0 0x7f042019ca70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f041800eaa0 tx=0x7f041800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041cff9640 1 -- 172.21.15.64:0/3034662117 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041800cdb0 con 0x7f04201067b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041cff9640 1 -- 172.21.15.64:0/3034662117 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0418005020 con 0x7f04201067b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f041cff9640 1 -- 172.21.15.64:0/3034662117 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04180116b0 con 0x7f04201067b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042019d440 con 0x7f04201067b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.518+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04201a4d20 con 0x7f04201067b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f041cff9640 1 -- 172.21.15.64:0/3034662117 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0418005560 con 0x7f04201067b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 Processor -- start 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 -- start start 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b21d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b2c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e399a87c60 con 0x55e399b222b0 2024-08-19T01:44:51.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55e399a1fb90 con 0x55e399b21990 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.521+0000 7f04276d8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55e399a83c20 con 0x55e399b2cb70 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b2c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b2c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54484/0 (socket says 172.21.15.64:54484) 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 -- 172.21.15.64:0/701400156 learned_addr learned my addr 172.21.15.64:0/701400156 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f0417fff640 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041cff9640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f040c0a2d70 0x7f040c0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f0417fff640 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b21d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 -- 172.21.15.64:0/701400156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 msgr2=0x55e399b2ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 -- 172.21.15.64:0/701400156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 msgr2=0x55e399b21d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b21d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 -- 172.21.15.64:0/701400156 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e399797580 con 0x55e399b222b0 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f0417fff640 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b21d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:44:51.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041f7fe640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f040c0a2d70 0x7f040c0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041effd640 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b2c630 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f041000bc20 tx=0x7f041000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6285005c89a31d8a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f041cff9640 1 -- 172.21.15.64:0/3034662117 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f04180cfae0 con 0x7f04201067b0 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f04157fa640 1 -- 172.21.15.64:0/701400156 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410015070 con 0x55e399b222b0 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f04157fa640 1 -- 172.21.15.64:0/701400156 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0410005500 con 0x55e399b222b0 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.522+0000 7f04157fa640 1 -- 172.21.15.64:0/701400156 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04100116a0 con 0x55e399b222b0 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 -- 172.21.15.64:0/701400156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 msgr2=0x55e399b2c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f041f7fe640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f040c0a2d70 0x7f040c0a5230 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0418097ff0 tx=0x7f041806db50 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b2c630 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f041000bc20 tx=0x7f041000f730 comp rx=0 tx=0).stop 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 -- 172.21.15.64:0/701400156 shutdown_connections 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b2c630 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 --2- 172.21.15.64:0/701400156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b21d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 -- 172.21.15.64:0/701400156 >> 172.21.15.64:0/701400156 conn(0x55e399b1df20 msgr2=0x55e399b1e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:51.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 -- 172.21.15.64:0/701400156 shutdown_connections 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 -- 172.21.15.64:0/701400156 wait complete. 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.523+0000 7f04276d8740 1 -- start start 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 1 Processor -- start 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b29a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b29f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e399b2f740 con 0x55e399b222b0 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55e399a257c0 con 0x55e399b21990 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55e399b2f8c0 con 0x55e399b2cb70 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04276d8740 10 client.0 _ll_register_callbacks cb 0x55e399a2ef50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b29f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b29f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54498/0 (socket says 172.21.15.64:54498) 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 -- 172.21.15.64:0/1133756989 learned_addr learned my addr 172.21.15.64:0/1133756989 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:51.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f0417fff640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f0417fff640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b29a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 -- 172.21.15.64:0/1133756989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 msgr2=0x55e399b2a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 -- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 msgr2=0x55e399b29a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b29a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 -- 172.21.15.64:0/1133756989 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e399861960 con 0x55e399b222b0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f0417fff640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b29a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f041effd640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b29f50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f041000bfa0 tx=0x7f041000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6058ee7f5becef6 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.524+0000 7f04067fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04067fc640 1 -- 172.21.15.64:0/1133756989 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410015030 con 0x55e399b222b0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04067fc640 1 -- 172.21.15.64:0/1133756989 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0410004e80 con 0x55e399b222b0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0410005500 con 0x55e399b222b0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04067fc640 1 -- 172.21.15.64:0/1133756989 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f041002d050 con 0x55e399b222b0 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 10 client.15945 fetch_fsmap learned FSMap version 29 2024-08-19T01:44:51.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e399b3b6a0 con 0x55e399b222b0 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04067fc640 1 -- 172.21.15.64:0/1133756989 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0410005500 con 0x55e399b222b0 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04067fc640 1 -- 172.21.15.64:0/1133756989 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0410011c50 con 0x55e399b222b0 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 10 client.15945 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 10 client.15945 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 20 client.15945 populate_metadata read hostname 'smithi064' 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399b3cb60 0x55e399b3ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 4 client.15945 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.525+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e399a951d0 con 0x55e399b3cb60 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.526+0000 7f0417fff640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399b3cb60 0x55e399b3ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.526+0000 7f04067fc640 10 client.15945 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.526+0000 7f0417fff640 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399b3cb60 0x55e399b3ef50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc6eceddebe6d8c5 server_cookie=f42b341bb02b30b0 in_seq=0 out_seq=0 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04067fc640 1 -- 172.21.15.64:0/1133756989 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e399a951d0 con 0x55e399b3cb60 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04067fc640 10 client.15945 handle_command_reply: tid=0 2024-08-19T01:44:51.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 client.15945 shutdown 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 msgr2=0x55e399b29f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b29f50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f041000bfa0 tx=0x7f041000bfd0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 shutdown_connections 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b2cb70 0x55e399b2a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b29f50 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399b3cb60 0x55e399b3ef50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 --2- 172.21.15.64:0/1133756989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b21990 0x55e399b29a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04067fc640 0 client.15945 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 >> 172.21.15.64:0/1133756989 conn(0x55e399b1df20 msgr2=0x55e399b1d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 shutdown_connections 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 1 -- 172.21.15.64:0/1133756989 wait complete. 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.527+0000 7f04276d8740 20 client.15945 trim_cache size 0 max 16384 2024-08-19T01:44:51.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 Processor -- start 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 -- start start 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b21990 0x55e399b243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b222b0 0x55e399b38220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 0x55e399b26c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04201b17c0 con 0x55e399b21990 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04201af090 con 0x55e399b2cb70 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f04276d8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04201acf00 con 0x55e399b222b0 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f0417fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b21990 0x55e399b243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f0417fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b21990 0x55e399b243f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54504/0 (socket says 172.21.15.64:54504) 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.544+0000 7f0417fff640 1 -- 172.21.15.64:0/9867319 learned_addr learned my addr 172.21.15.64:0/9867319 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0417fff640 1 -- 172.21.15.64:0/9867319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b222b0 msgr2=0x55e399b38220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0417fff640 1 --2- 172.21.15.64:0/9867319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b222b0 0x55e399b38220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0417fff640 1 -- 172.21.15.64:0/9867319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 msgr2=0x55e399b26c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0417fff640 1 --2- 172.21.15.64:0/9867319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 0x55e399b26c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0417fff640 1 -- 172.21.15.64:0/9867319 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e399b1d9c0 con 0x55e399b21990 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0417fff640 1 --2- 172.21.15.64:0/9867319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b21990 0x55e399b243f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0408009900 tx=0x7f04080058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3054fa8f74c890db server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0414ff9640 1 -- 172.21.15.64:0/9867319 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408035040 con 0x55e399b21990 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0414ff9640 1 -- 172.21.15.64:0/9867319 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0408005500 con 0x55e399b21990 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.545+0000 7f0414ff9640 1 -- 172.21.15.64:0/9867319 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408004e00 con 0x55e399b21990 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- 172.21.15.64:0/9867319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b21990 msgr2=0x55e399b243f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 --2- 172.21.15.64:0/9867319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b21990 0x55e399b243f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0408009900 tx=0x7f04080058d0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- 172.21.15.64:0/9867319 shutdown_connections 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 --2- 172.21.15.64:0/9867319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 0x55e399b26c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 --2- 172.21.15.64:0/9867319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b222b0 0x55e399b38220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 --2- 172.21.15.64:0/9867319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b21990 0x55e399b243f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- 172.21.15.64:0/9867319 >> 172.21.15.64:0/9867319 conn(0x55e399b1df20 msgr2=0x55e399b37b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- 172.21.15.64:0/9867319 shutdown_connections 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- 172.21.15.64:0/9867319 wait complete. 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- start start 2024-08-19T01:44:51.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 Processor -- start 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b21990 0x55e399b3fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 0x55e399b40920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04201ad550 con 0x55e399b222b0 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04201ae9c0 con 0x55e399b2cb70 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.546+0000 7f04276d8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04201adc20 con 0x55e399b21990 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04276d8740 10 client.0 _ll_register_callbacks cb 0x7f04100056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54514/0 (socket says 172.21.15.64:54514) 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 -- 172.21.15.64:0/1339688137 learned_addr learned my addr 172.21.15.64:0/1339688137 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f0417fff640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b21990 0x55e399b3fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f0417fff640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 0x55e399b40920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 -- 172.21.15.64:0/1339688137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b21990 msgr2=0x55e399b3fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b21990 0x55e399b3fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 -- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 msgr2=0x55e399b40920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 0x55e399b40920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 -- 172.21.15.64:0/1339688137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e399ba65a0 con 0x55e399b222b0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f0417fff640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b21990 0x55e399b3fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f041effd640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b403e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x55e399b2b050 tx=0x7f041000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416f97020488f64c server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04067fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04067fc640 1 -- 172.21.15.64:0/1339688137 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410036060 con 0x55e399b222b0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0408005500 con 0x55e399b222b0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04067fc640 1 -- 172.21.15.64:0/1339688137 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0410005a70 con 0x55e399b222b0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04067fc640 1 -- 172.21.15.64:0/1339688137 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f041003d050 con 0x55e399b222b0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04276d8740 10 client.15951 fetch_fsmap learned FSMap version 29 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.547+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e399ba78f0 con 0x55e399b222b0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04067fc640 1 -- 172.21.15.64:0/1339688137 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0408005500 con 0x55e399b222b0 2024-08-19T01:44:51.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04067fc640 1 -- 172.21.15.64:0/1339688137 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0410011810 con 0x55e399b222b0 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04276d8740 10 client.15951 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04276d8740 10 client.15951 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04276d8740 20 client.15951 populate_metadata read hostname 'smithi064' 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04276d8740 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399ba80a0 0x55e399baa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04276d8740 4 client.15951 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2e2b1d45", "release": true, "await": true}] 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 40, 2024-08-19T01:44:51.579 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "2e2b1d45": { 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 40, 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 140.2, 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:44:51.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2e2b1d45", "release": true, "await": true}) -- 0x55e399b191d0 con 0x55e399ba80a0 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.548+0000 7f0417fff640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399ba80a0 0x55e399baa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.549+0000 7f04067fc640 10 client.15951 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:51.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.549+0000 7f0417fff640 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399ba80a0 0x55e399baa4c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=232112b41423b55 server_cookie=cc16f8f6400d3af1 in_seq=0 out_seq=0 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.556+0000 7f04067fc640 1 -- 172.21.15.64:0/1339688137 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e399b191d0 con 0x55e399ba80a0 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.556+0000 7f04067fc640 10 client.15951 handle_command_reply: tid=0 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.556+0000 7f04276d8740 1 client.15951 shutdown 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.556+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 msgr2=0x55e399b403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.556+0000 7f04276d8740 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b403e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x55e399b2b050 tx=0x7f041000b8f0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 shutdown_connections 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55e399ba80a0 0x55e399baa4c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e399b2cb70 0x55e399b40920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e399b222b0 0x55e399b403e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04067fc640 0 client.15951 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 --2- 172.21.15.64:0/1339688137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e399b21990 0x55e399b3fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 >> 172.21.15.64:0/1339688137 conn(0x55e399b1df20 msgr2=0x55e399b1d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:51.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 shutdown_connections 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 1 -- 172.21.15.64:0/1339688137 wait complete. 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04276d8740 20 client.15951 trim_cache size 0 max 16384 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f040c0a2d70 msgr2=0x7f040c0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f040c0a2d70 0x7f040c0a5230 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0418097ff0 tx=0x7f041806db50 comp rx=0 tx=0).stop 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201067b0 msgr2=0x7f042019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201067b0 0x7f042019ca70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f041800eaa0 tx=0x7f041800ef60 comp rx=0 tx=0).stop 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 shutdown_connections 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f040c0a2d70 0x7f040c0a5230 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04201067b0 0x7f042019ca70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0420101150 0x7f04201986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 --2- 172.21.15.64:0/3034662117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04201007b0 0x7f04201981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 >> 172.21.15.64:0/3034662117 conn(0x7f04200fc4d0 msgr2=0x7f04201045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 shutdown_connections 2024-08-19T01:44:51.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:51.557+0000 7f04252da640 1 -- 172.21.15.64:0/3034662117 wait complete. 2024-08-19T01:44:52.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1012576 8.19 MB/sec execute 504 sec latency 19935.340 ms 2024-08-19T01:44:52.464 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2e2b1d45', total seconds elapsed: 23.6 2024-08-19T01:44:52.464 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 20.15 2024-08-19T01:44:52.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:44:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:44:53.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:53 smithi064 ceph-mon[18515]: pgmap v362: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 -- 172.21.15.64:0/1026829061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 msgr2=0x7f33a011d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1026829061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a011d260 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3398009be0 tx=0x7f339802a6c0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 -- 172.21.15.64:0/1026829061 shutdown_connections 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1026829061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a011d260 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1026829061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a0109560 0x7f33a01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1026829061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a0109180 0x7f33a010ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 -- 172.21.15.64:0/1026829061 >> 172.21.15.64:0/1026829061 conn(0x7f33a006de70 msgr2=0x7f33a006e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 -- 172.21.15.64:0/1026829061 shutdown_connections 2024-08-19T01:44:53.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.371+0000 7f33a71f4640 1 -- 172.21.15.64:0/1026829061 wait complete. 2024-08-19T01:44:53.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.372+0000 7f33a71f4640 1 Processor -- start 2024-08-19T01:44:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a71f4640 1 -- start start 2024-08-19T01:44:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a71f4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a0109180 0x7f33a01a7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a71f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a0109560 0x7f33a01a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a71f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a01ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a71f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f33a0120ad0 con 0x7f33a011ae70 2024-08-19T01:44:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a71f4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f33a0120950 con 0x7f33a0109180 2024-08-19T01:44:53.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a71f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f33a0120c50 con 0x7f33a0109560 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a5791640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a01ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a5791640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a01ab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54544/0 (socket says 172.21.15.64:54544) 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a5791640 1 -- 172.21.15.64:0/1975846468 learned_addr learned my addr 172.21.15.64:0/1975846468 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f33a4f90640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a0109180 0x7f33a01a7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.374+0000 7f339ffff640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a0109560 0x7f33a01a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.375+0000 7f33a5791640 1 -- 172.21.15.64:0/1975846468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a0109560 msgr2=0x7f33a01a7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.375+0000 7f33a5791640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a0109560 0x7f33a01a7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.375+0000 7f33a5791640 1 -- 172.21.15.64:0/1975846468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a0109180 msgr2=0x7f33a01a7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.375+0000 7f33a5791640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a0109180 0x7f33a01a7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.375+0000 7f33a5791640 1 -- 172.21.15.64:0/1975846468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a01abfe0 con 0x7f33a011ae70 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.375+0000 7f33a4f90640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a0109180 0x7f33a01a7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:53.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.375+0000 7f33a5791640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a01ab900 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3398031610 tx=0x7f3398030c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:53.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.376+0000 7f339dffb640 1 -- 172.21.15.64:0/1975846468 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398052020 con 0x7f33a011ae70 2024-08-19T01:44:53.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.376+0000 7f339dffb640 1 -- 172.21.15.64:0/1975846468 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3398003970 con 0x7f33a011ae70 2024-08-19T01:44:53.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.376+0000 7f339dffb640 1 -- 172.21.15.64:0/1975846468 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398036040 con 0x7f33a011ae70 2024-08-19T01:44:53.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.376+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a01ac270 con 0x7f33a011ae70 2024-08-19T01:44:53.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.376+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a01ac640 con 0x7f33a011ae70 2024-08-19T01:44:53.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.378+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a01a8390 con 0x7f33a011ae70 2024-08-19T01:44:53.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.380+0000 7f339dffb640 1 -- 172.21.15.64:0/1975846468 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3398055020 con 0x7f33a011ae70 2024-08-19T01:44:53.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.381+0000 7f339dffb640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f33880a2dc0 0x7f33880a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:53.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.381+0000 7f339dffb640 1 -- 172.21.15.64:0/1975846468 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f33981014c0 con 0x7f33a011ae70 2024-08-19T01:44:53.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.382+0000 7f33a4f90640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f33880a2dc0 0x7f33880a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:53.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.383+0000 7f33a4f90640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f33880a2dc0 0x7f33880a5280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f339000bfd0 tx=0x7f339000b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:53.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.389+0000 7f339dffb640 1 -- 172.21.15.64:0/1975846468 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f339805c020 con 0x7f33a011ae70 2024-08-19T01:44:53.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1013360 8.17 MB/sec execute 505 sec latency 49.269 ms 2024-08-19T01:44:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:53 smithi123 ceph-mon[21829]: pgmap v362: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-08-19T01:44:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:53 smithi161 ceph-mon[21757]: pgmap v362: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-08-19T01:44:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.622+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33a002d5e0 con 0x7f33a011ae70 2024-08-19T01:44:53.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.623+0000 7f339dffb640 1 -- 172.21.15.64:0/1975846468 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7f33980cdcd0 con 0x7f33a011ae70 2024-08-19T01:44:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:44:53.624 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:44:53.624 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f33880a2dc0 msgr2=0x7f33880a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f33880a2dc0 0x7f33880a5280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f339000bfd0 tx=0x7f339000b390 comp rx=0 tx=0).stop 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 msgr2=0x7f33a01ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a01ab900 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3398031610 tx=0x7f3398030c60 comp rx=0 tx=0).stop 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 shutdown_connections 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f33880a2dc0 0x7f33880a5280 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a011ae70 0x7f33a01ab900 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a0109560 0x7f33a01a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 --2- 172.21.15.64:0/1975846468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a0109180 0x7f33a01a7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 >> 172.21.15.64:0/1975846468 conn(0x7f33a006de70 msgr2=0x7f33a011bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 shutdown_connections 2024-08-19T01:44:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:53.625+0000 7f33a71f4640 1 -- 172.21.15.64:0/1975846468 wait complete. 2024-08-19T01:44:54.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1016049 8.19 MB/sec execute 506 sec latency 15.684 ms 2024-08-19T01:44:54.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:54 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1975846468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:54.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:54 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1975846468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:54.471 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:54 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1975846468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:44:54.499 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0' 2024-08-19T01:44:54.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 140.15 --await-for 45.0 2024-08-19T01:44:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:44:55.158 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:55 smithi064 ceph-mon[18515]: pgmap v363: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-08-19T01:44:55.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/1555367817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b441031e0 msgr2=0x7f0b4410b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/1555367817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b441031e0 0x7f0b4410b6d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009860 tx=0x7f0b3402a3d0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/1555367817 shutdown_connections 2024-08-19T01:44:55.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/1555367817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b4410bcf0 0x7f0b4410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/1555367817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b441031e0 0x7f0b4410b6d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/1555367817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 0x7f0b44102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/1555367817 >> 172.21.15.64:0/1555367817 conn(0x7f0b440fc340 msgr2=0x7f0b440fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:55.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/1555367817 shutdown_connections 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.284+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/1555367817 wait complete. 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.285+0000 7f0b4ae8f640 1 Processor -- start 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.285+0000 7f0b4ae8f640 1 -- start start 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4ae8f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 0x7f0b4419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4ae8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b4410bcf0 0x7f0b4410aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4ae8f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b4410afe0 0x7f0b4410b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4ae8f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0b44111b40 con 0x7f0b441028c0 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4ae8f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0b441119c0 con 0x7f0b4410afe0 2024-08-19T01:44:55.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4ae8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b44111cc0 con 0x7f0b4410bcf0 2024-08-19T01:44:55.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 0x7f0b4419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4a68e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b4410afe0 0x7f0b4410b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4a68e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b4410afe0 0x7f0b4410b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:51430/0 (socket says 172.21.15.64:51430) 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 0x7f0b4419eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54562/0 (socket says 172.21.15.64:54562) 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4a68e640 1 -- 172.21.15.64:0/3358975923 learned_addr learned my addr 172.21.15.64:0/3358975923 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4968c640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b4410bcf0 0x7f0b4410aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b49e8d640 1 -- 172.21.15.64:0/3358975923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b4410bcf0 msgr2=0x7f0b4410aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b4410bcf0 0x7f0b4410aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b49e8d640 1 -- 172.21.15.64:0/3358975923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b4410afe0 msgr2=0x7f0b4410b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b4410afe0 0x7f0b4410b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b49e8d640 1 -- 172.21.15.64:0/3358975923 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4419a8f0 con 0x7f0b441028c0 2024-08-19T01:44:55.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.286+0000 7f0b4968c640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b4410bcf0 0x7f0b4410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:55.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.287+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 0x7f0b4419eea0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00ef90 tx=0x7f0b3c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:55.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.287+0000 7f0b3affd640 1 -- 172.21.15.64:0/3358975923 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c01b070 con 0x7f0b441028c0 2024-08-19T01:44:55.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.287+0000 7f0b3affd640 1 -- 172.21.15.64:0/3358975923 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b3c00bb10 con 0x7f0b441028c0 2024-08-19T01:44:55.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.287+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4419abe0 con 0x7f0b441028c0 2024-08-19T01:44:55.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.287+0000 7f0b3affd640 1 -- 172.21.15.64:0/3358975923 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c016c00 con 0x7f0b441028c0 2024-08-19T01:44:55.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.287+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4419b0e0 con 0x7f0b441028c0 2024-08-19T01:44:55.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.290+0000 7f0b3affd640 1 -- 172.21.15.64:0/3358975923 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0b3c01e070 con 0x7f0b441028c0 2024-08-19T01:44:55.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b3affd640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0b300a2d70 0x7f0b300a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 Processor -- start 2024-08-19T01:44:55.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4968c640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0b300a2d70 0x7f0b300a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b3affd640 1 -- 172.21.15.64:0/3358975923 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0b3c0cf890 con 0x7f0b441028c0 2024-08-19T01:44:55.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 -- start start 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7268490 0x559de7268870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7273130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7273670 0x559de7275a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559de71cd810 con 0x559de7273670 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559de7174c70 con 0x559de7268490 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559de71c97d0 con 0x559de7268db0 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7268490 0x559de7268870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b4a68e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7273670 0x559de7275a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7268490 0x559de7268870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:51438/0 (socket says 172.21.15.64:51438) 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.291+0000 7f0b49e8d640 1 -- 172.21.15.64:0/806822129 learned_addr learned my addr 172.21.15.64:0/806822129 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4968c640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0b300a2d70 0x7f0b300a5230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009830 tx=0x7f0b34005c90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4968c640 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7273130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4a68e640 1 -- 172.21.15.64:0/806822129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 msgr2=0x559de7273130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7273130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4a68e640 1 -- 172.21.15.64:0/806822129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7268490 msgr2=0x559de7268870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7268490 0x559de7268870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4a68e640 1 -- 172.21.15.64:0/806822129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de6f3fba0 con 0x559de7273670 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4968c640 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7273130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7273670 0x559de7275a60 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009c90 tx=0x7f0b40011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becc68d4c0be1d79 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b2ffff640 1 -- 172.21.15.64:0/806822129 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4001e070 con 0x559de7273670 2024-08-19T01:44:55.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b2ffff640 1 -- 172.21.15.64:0/806822129 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b40009e40 con 0x559de7273670 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.292+0000 7f0b2ffff640 1 -- 172.21.15.64:0/806822129 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40018790 con 0x559de7273670 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.293+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/806822129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7273670 msgr2=0x559de7275a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.293+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7273670 0x559de7275a60 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009c90 tx=0x7f0b40011810 comp rx=0 tx=0).stop 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.293+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/806822129 shutdown_connections 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.293+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7273670 0x559de7275a60 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.293+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7273130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.293+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/806822129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7268490 0x559de7268870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.293+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/806822129 >> 172.21.15.64:0/806822129 conn(0x559de7264a20 msgr2=0x559de7264e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.294+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/806822129 shutdown_connections 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.294+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/806822129 wait complete. 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.294+0000 7f0b4e4ef740 1 -- start start 2024-08-19T01:44:55.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.294+0000 7f0b4e4ef740 1 Processor -- start 2024-08-19T01:44:55.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de7270500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de7270a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de7270f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559de7276240 con 0x559de7268db0 2024-08-19T01:44:55.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de7270500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4968c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de7270a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de7270500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:34122/0 (socket says 172.21.15.64:34122) 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 -- 172.21.15.64:0/2059653828 learned_addr learned my addr 172.21.15.64:0/2059653828 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559de71889a0 con 0x559de7273670 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559de72763c0 con 0x559de7268490 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 -- 172.21.15.64:0/2059653828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 msgr2=0x559de7270f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de7270f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 -- 172.21.15.64:0/2059653828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 msgr2=0x559de7270a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de7270a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b49e8d640 1 -- 172.21.15.64:0/2059653828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de7194220 con 0x559de7268490 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4968c640 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de7270a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.295+0000 7f0b4e4ef740 10 client.0 _ll_register_callbacks cb 0x559de7176c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de7270500 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c09b750 tx=0x7f0b3c09b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=816ab8b1855e881a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:55.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559de71daf40 con 0x559de7268490 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/2059653828 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c0a4610 con 0x559de7268490 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/2059653828 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0b3c0980c0 con 0x559de7268490 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/2059653828 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b3c0b4ba0 con 0x559de7268490 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b4e4ef740 10 client.25402 fetch_fsmap learned FSMap version 29 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.296+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({fsmap=29}) -- 0x559de7281d00 con 0x559de7268490 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/2059653828 <== mon.1 v2:172.21.15.161:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0b3c00b040 con 0x559de7268490 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4e4ef740 10 client.25402 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4e4ef740 10 client.25402 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4e4ef740 20 client.25402 populate_metadata read hostname 'smithi064' 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de7285620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4e4ef740 4 client.25402 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559de71daf40 con 0x559de7283190 2024-08-19T01:44:55.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4968c640 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de7285620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b4968c640 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de7285620 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95f17c926a8f5f32 server_cookie=2464c7f017724401 in_seq=0 out_seq=0 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.297+0000 7f0b2d7fa640 10 client.25402 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/2059653828 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559de71daf40 con 0x559de7283190 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b2d7fa640 10 client.25402 handle_command_reply: tid=0 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 client.25402 shutdown 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 msgr2=0x559de7270500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de7270500 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c09b750 tx=0x7f0b3c09b650 comp rx=0 tx=0).stop 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 shutdown_connections 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de7285620 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de7270f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b2d7fa640 0 client.25402 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de7270a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/2059653828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de7270500 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.298+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 >> 172.21.15.64:0/2059653828 conn(0x559de7264a20 msgr2=0x559de7263d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:55.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.299+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 shutdown_connections 2024-08-19T01:44:55.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.299+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/2059653828 wait complete. 2024-08-19T01:44:55.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.299+0000 7f0b4e4ef740 20 client.25402 trim_cache size 0 max 16384 2024-08-19T01:44:55.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.318+0000 7f0b4e4ef740 1 Processor -- start 2024-08-19T01:44:55.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.318+0000 7f0b4e4ef740 1 -- start start 2024-08-19T01:44:55.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.318+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268490 0x559de726aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7270dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de728aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0b441c09f0 con 0x559de7268490 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0b441be2c0 con 0x559de7273670 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b441bc130 con 0x559de7268db0 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268490 0x559de726aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4968c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7270dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b49e8d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268490 0x559de726aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54604/0 (socket says 172.21.15.64:54604) 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4968c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7270dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:34138/0 (socket says 172.21.15.64:34138) 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b49e8d640 1 -- 172.21.15.64:0/1680495325 learned_addr learned my addr 172.21.15.64:0/1680495325 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4a68e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de728aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4a68e640 1 -- 172.21.15.64:0/1680495325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 msgr2=0x559de7270dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/1680495325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7270dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4a68e640 1 -- 172.21.15.64:0/1680495325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268490 msgr2=0x559de726aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/1680495325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268490 0x559de726aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4a68e640 1 -- 172.21.15.64:0/1680495325 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de7264850 con 0x559de7273670 2024-08-19T01:44:55.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.319+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/1680495325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de728aef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009c60 tx=0x7f0b40009a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30236758081e9c1d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:55.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.320+0000 7f0b2f7fe640 1 -- 172.21.15.64:0/1680495325 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40028020 con 0x559de7273670 2024-08-19T01:44:55.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.320+0000 7f0b2f7fe640 1 -- 172.21.15.64:0/1680495325 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b400050c0 con 0x559de7273670 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.320+0000 7f0b2f7fe640 1 -- 172.21.15.64:0/1680495325 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4001e040 con 0x559de7273670 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.320+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/1680495325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 msgr2=0x559de728aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.320+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/1680495325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de728aef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009c60 tx=0x7f0b40009a40 comp rx=0 tx=0).stop 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/1680495325 shutdown_connections 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/1680495325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de728aef0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/1680495325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268db0 0x559de7270dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/1680495325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268490 0x559de726aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/1680495325 >> 172.21.15.64:0/1680495325 conn(0x559de7264a20 msgr2=0x559de7263d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/1680495325 shutdown_connections 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/1680495325 wait complete. 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 -- start start 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.321+0000 7f0b4e4ef740 1 Processor -- start 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de727ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de727bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4e4ef740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de726d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0b441bc780 con 0x559de7268db0 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0b441bdbf0 con 0x559de7273670 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4e4ef740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b441bce50 con 0x559de7268490 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4e4ef740 10 client.0 _ll_register_callbacks cb 0x7f0b3c00eee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4968c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de727bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4a68e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de726d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4968c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de727bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54614/0 (socket says 172.21.15.64:54614) 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4a68e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de726d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:51454/0 (socket says 172.21.15.64:51454) 2024-08-19T01:44:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b4968c640 1 -- 172.21.15.64:0/4257733010 learned_addr learned my addr 172.21.15.64:0/4257733010 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:44:55.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.322+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de727ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b4a68e640 1 -- 172.21.15.64:0/4257733010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 msgr2=0x559de727ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de727ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b4a68e640 1 -- 172.21.15.64:0/4257733010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 msgr2=0x559de727bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de727bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b4a68e640 1 -- 172.21.15.64:0/4257733010 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de726d940 con 0x559de7273670 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b4a68e640 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de726d110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009ad0 tx=0x7f0b40004930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=344ada126b3b39c9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/4257733010 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40005480 con 0x559de7273670 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/4257733010 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f0b40004a30 con 0x559de7273670 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.323+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b400050c0 con 0x559de7273670 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/4257733010 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b40004c50 con 0x559de7273670 2024-08-19T01:44:55.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 10 client.25409 fetch_fsmap learned FSMap version 29 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559de72edb70 con 0x559de7273670 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/4257733010 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b400050c0 con 0x559de7273670 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/4257733010 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0b40019410 con 0x559de7273670 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 10 client.25409 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 10 client.25409 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 20 client.25409 populate_metadata read hostname 'smithi064' 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de72f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 4 client.25409 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}] 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.324+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 140.15, "await_for": 45.0}) -- 0x559de71d5290 con 0x559de7283190 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.325+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de72f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.325+0000 7f0b49e8d640 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de72f00c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=825e7b8c088aed7 server_cookie=788a367591d3a25a in_seq=0 out_seq=0 2024-08-19T01:44:55.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.325+0000 7f0b2d7fa640 10 client.25409 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b2d7fa640 1 -- 172.21.15.64:0/4257733010 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559de71d5290 con 0x559de7283190 2024-08-19T01:44:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b2d7fa640 10 client.25409 handle_command_reply: tid=0 2024-08-19T01:44:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 client.25409 shutdown 2024-08-19T01:44:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 msgr2=0x559de726d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de726d110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009ad0 tx=0x7f0b40004930 comp rx=0 tx=0).stop 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 shutdown_connections 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559de7283190 0x559de72f00c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559de7273670 0x559de726d110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559de7268db0 0x559de727bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b2d7fa640 0 client.25409 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 --2- 172.21.15.64:0/4257733010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559de7268490 0x559de727ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 >> 172.21.15.64:0/4257733010 conn(0x559de7264a20 msgr2=0x559de727e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 shutdown_connections 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 1 -- 172.21.15.64:0/4257733010 wait complete. 2024-08-19T01:44:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.384+0000 7f0b4e4ef740 20 client.25409 trim_cache size 0 max 16384 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 42, 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "1a8e1bbc": { 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 42, 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:44:55.387 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 140.2, 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:55.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0b300a2d70 msgr2=0x7f0b300a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0b300a2d70 0x7f0b300a5230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009830 tx=0x7f0b34005c90 comp rx=0 tx=0).stop 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 msgr2=0x7f0b4419eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 0x7f0b4419eea0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00ef90 tx=0x7f0b3c00c800 comp rx=0 tx=0).stop 2024-08-19T01:44:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 shutdown_connections 2024-08-19T01:44:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0b300a2d70 0x7f0b300a5230 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b4410afe0 0x7f0b4410b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b4410bcf0 0x7f0b4410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 --2- 172.21.15.64:0/3358975923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b441028c0 0x7f0b4419eea0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:44:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 >> 172.21.15.64:0/3358975923 conn(0x7f0b440fc340 msgr2=0x7f0b44100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:44:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 shutdown_connections 2024-08-19T01:44:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:44:55.385+0000 7f0b4ae8f640 1 -- 172.21.15.64:0/3358975923 wait complete. 2024-08-19T01:44:55.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.18 MB/sec execute 507 sec latency 49.337 ms 2024-08-19T01:44:55.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:55 smithi123 ceph-mon[21829]: pgmap v363: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-08-19T01:44:55.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:55 smithi161 ceph-mon[21757]: pgmap v363: 129 pgs: 129 active+clean; 580 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-08-19T01:44:56.203 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1a8e1bbc', quiesce took 3.7 seconds. Will release after: 16.45 2024-08-19T01:44:56.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.17 MB/sec execute 508 sec latency 1024.366 ms 2024-08-19T01:44:57.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.15 MB/sec execute 509 sec latency 2024.463 ms 2024-08-19T01:44:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:57 smithi123 ceph-mon[21829]: pgmap v364: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 133 op/s 2024-08-19T01:44:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:57 smithi161 ceph-mon[21757]: pgmap v364: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 133 op/s 2024-08-19T01:44:57.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:57 smithi064 ceph-mon[18515]: pgmap v364: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 133 op/s 2024-08-19T01:44:58.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.13 MB/sec execute 510 sec latency 3024.539 ms 2024-08-19T01:44:59.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.12 MB/sec execute 511 sec latency 4024.630 ms 2024-08-19T01:44:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:44:59 smithi123 ceph-mon[21829]: pgmap v365: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 113 op/s 2024-08-19T01:44:59.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:44:59 smithi161 ceph-mon[21757]: pgmap v365: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 113 op/s 2024-08-19T01:44:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:44:59 smithi064 ceph-mon[18515]: pgmap v365: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 113 op/s 2024-08-19T01:45:00.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.10 MB/sec execute 512 sec latency 5024.754 ms 2024-08-19T01:45:01.410 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.09 MB/sec execute 513 sec latency 6024.863 ms 2024-08-19T01:45:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:01 smithi123 ceph-mon[21829]: pgmap v366: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 139 op/s 2024-08-19T01:45:01.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:01 smithi161 ceph-mon[21757]: pgmap v366: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 139 op/s 2024-08-19T01:45:01.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:01 smithi064 ceph-mon[18515]: pgmap v366: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 139 op/s 2024-08-19T01:45:02.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.07 MB/sec execute 514 sec latency 7024.959 ms 2024-08-19T01:45:03.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.06 MB/sec execute 515 sec latency 8025.004 ms 2024-08-19T01:45:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:03 smithi123 ceph-mon[21829]: pgmap v367: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-08-19T01:45:03.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:03 smithi161 ceph-mon[21757]: pgmap v367: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-08-19T01:45:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:03 smithi064 ceph-mon[18515]: pgmap v367: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-08-19T01:45:04.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.04 MB/sec execute 516 sec latency 9025.108 ms 2024-08-19T01:45:05.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.02 MB/sec execute 517 sec latency 10025.180 ms 2024-08-19T01:45:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:05 smithi123 ceph-mon[21829]: pgmap v368: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-08-19T01:45:05.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:05 smithi161 ceph-mon[21757]: pgmap v368: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-08-19T01:45:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:05 smithi064 ceph-mon[18515]: pgmap v368: 129 pgs: 129 active+clean; 595 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 96 op/s 2024-08-19T01:45:06.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 8.01 MB/sec execute 518 sec latency 11025.277 ms 2024-08-19T01:45:07.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.99 MB/sec execute 519 sec latency 12025.331 ms 2024-08-19T01:45:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:07 smithi123 ceph-mon[21829]: pgmap v369: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-08-19T01:45:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:07 smithi161 ceph-mon[21757]: pgmap v369: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-08-19T01:45:07.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:07 smithi064 ceph-mon[18515]: pgmap v369: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-08-19T01:45:08.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.98 MB/sec execute 520 sec latency 13025.379 ms 2024-08-19T01:45:08.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:08 smithi123 ceph-mon[21829]: osdmap e105: 12 total, 12 up, 12 in 2024-08-19T01:45:08.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:08 smithi161 ceph-mon[21757]: osdmap e105: 12 total, 12 up, 12 in 2024-08-19T01:45:08.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:08 smithi064 ceph-mon[18515]: osdmap e105: 12 total, 12 up, 12 in 2024-08-19T01:45:09.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.96 MB/sec execute 521 sec latency 14025.457 ms 2024-08-19T01:45:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:09 smithi123 ceph-mon[21829]: pgmap v371: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-08-19T01:45:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:09 smithi161 ceph-mon[21757]: pgmap v371: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-08-19T01:45:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:09 smithi064 ceph-mon[18515]: pgmap v371: 129 pgs: 129 active+clean; 589 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-08-19T01:45:10.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.95 MB/sec execute 522 sec latency 15025.559 ms 2024-08-19T01:45:11.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.93 MB/sec execute 523 sec latency 16025.653 ms 2024-08-19T01:45:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:11 smithi123 ceph-mon[21829]: pgmap v372: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 796 KiB/s wr, 62 op/s 2024-08-19T01:45:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:11 smithi161 ceph-mon[21757]: pgmap v372: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 796 KiB/s wr, 62 op/s 2024-08-19T01:45:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:11 smithi064 ceph-mon[18515]: pgmap v372: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 796 KiB/s wr, 62 op/s 2024-08-19T01:45:12.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.92 MB/sec execute 524 sec latency 17025.763 ms 2024-08-19T01:45:12.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:45:12.924 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:45:13.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:13 smithi064 ceph-mon[18515]: pgmap v373: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 796 KiB/s wr, 62 op/s 2024-08-19T01:45:13.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.90 MB/sec execute 525 sec latency 18025.841 ms 2024-08-19T01:45:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:13 smithi123 ceph-mon[21829]: pgmap v373: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 796 KiB/s wr, 62 op/s 2024-08-19T01:45:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:13 smithi161 ceph-mon[21757]: pgmap v373: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 796 KiB/s wr, 62 op/s 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 -- 172.21.15.64:0/810277147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 msgr2=0x7fedf010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/810277147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf010d770 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fede0009ca0 tx=0x7fede002a950 comp rx=0 tx=0).stop 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 -- 172.21.15.64:0/810277147 shutdown_connections 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/810277147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf010d770 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/810277147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf01022a0 0x7fedf01026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/810277147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedf0100b50 0x7fedf0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 -- 172.21.15.64:0/810277147 >> 172.21.15.64:0/810277147 conn(0x7fedf00fc500 msgr2=0x7fedf00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:13.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 -- 172.21.15.64:0/810277147 shutdown_connections 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.533+0000 7fedf4cbe640 1 -- 172.21.15.64:0/810277147 wait complete. 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 Processor -- start 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 -- start start 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0100b50 0x7fedf0198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedf01022a0 0x7fedf0198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedf0110cb0 con 0x7fedf0102c20 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fedf0110b30 con 0x7fedf0100b50 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedf4cbe640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fedf0110e30 con 0x7fedf01022a0 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf019ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55490/0 (socket says 172.21.15.64:55490) 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 -- 172.21.15.64:0/1939903830 learned_addr learned my addr 172.21.15.64:0/1939903830 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedee59c640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0100b50 0x7fedf0198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fededd9b640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedf01022a0 0x7fedf0198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 -- 172.21.15.64:0/1939903830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedf01022a0 msgr2=0x7fedf0198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedf01022a0 0x7fedf0198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 -- 172.21.15.64:0/1939903830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0100b50 msgr2=0x7fedf0198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0100b50 0x7fedf0198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.534+0000 7fedeed9d640 1 -- 172.21.15.64:0/1939903830 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf019d0e0 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.535+0000 7fedeed9d640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf019ca00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fede002ae70 tx=0x7fede002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.535+0000 7fede77fe640 1 -- 172.21.15.64:0/1939903830 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede0034040 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.535+0000 7fede77fe640 1 -- 172.21.15.64:0/1939903830 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede0030920 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.535+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf019d370 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.535+0000 7fede77fe640 1 -- 172.21.15.64:0/1939903830 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede00047f0 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.535+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf00fc8e0 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.536+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf0101ee0 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.537+0000 7fede77fe640 1 -- 172.21.15.64:0/1939903830 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fede0037020 con 0x7fedf0102c20 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.538+0000 7fede77fe640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feddc0a2cf0 0x7feddc0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.538+0000 7fedee59c640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feddc0a2cf0 0x7feddc0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.538+0000 7fede77fe640 1 -- 172.21.15.64:0/1939903830 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9642+0+0 (secure 0 0 0) 0x7fede00f0670 con 0x7fedf0102c20 2024-08-19T01:45:13.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.538+0000 7fedee59c640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feddc0a2cf0 0x7feddc0a51b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fede8005fd0 tx=0x7fede800b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:13.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.541+0000 7fede77fe640 1 -- 172.21.15.64:0/1939903830 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fede00bcfd0 con 0x7fedf0102c20 2024-08-19T01:45:13.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.756+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedf0100f30 con 0x7fedf0102c20 2024-08-19T01:45:13.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.758+0000 7fede77fe640 1 -- 172.21.15.64:0/1939903830 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11921 (secure 0 0 0) 0x7fedf0100f30 con 0x7fedf0102c20 2024-08-19T01:45:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:45:13.759 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":29,"btime":"2024-08-19T01:37:11:958396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:37:11.550080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24436,24452,24476]},"id":1}]} 2024-08-19T01:45:13.759 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 29 2024-08-19T01:45:13.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feddc0a2cf0 msgr2=0x7feddc0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:13.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feddc0a2cf0 0x7feddc0a51b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fede8005fd0 tx=0x7fede800b390 comp rx=0 tx=0).stop 2024-08-19T01:45:13.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 msgr2=0x7fedf019ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:13.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf019ca00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fede002ae70 tx=0x7fede002f5d0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 shutdown_connections 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feddc0a2cf0 0x7feddc0a51b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedf0102c20 0x7fedf019ca00 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedf01022a0 0x7fedf0198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 --2- 172.21.15.64:0/1939903830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0100b50 0x7fedf0198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 >> 172.21.15.64:0/1939903830 conn(0x7fedf00fc500 msgr2=0x7fedf0069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 shutdown_connections 2024-08-19T01:45:13.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:13.760+0000 7fedf4cbe640 1 -- 172.21.15.64:0/1939903830 wait complete. 2024-08-19T01:45:14.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017099 7.89 MB/sec execute 526 sec latency 19025.896 ms 2024-08-19T01:45:14.434 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 1a8e1bbc --release --await' 2024-08-19T01:45:14.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 1a8e1bbc --release --await 2024-08-19T01:45:14.692 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:45:15.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:14 smithi064 ceph-mon[18515]: osdmap e106: 12 total, 12 up, 12 in 2024-08-19T01:45:15.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:14 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1939903830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:45:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:14 smithi123 ceph-mon[21829]: osdmap e106: 12 total, 12 up, 12 in 2024-08-19T01:45:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:14 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1939903830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:45:15.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:14 smithi161 ceph-mon[21757]: osdmap e106: 12 total, 12 up, 12 in 2024-08-19T01:45:15.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:14 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1939903830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:45:15.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 -- 172.21.15.64:0/2860306356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 msgr2=0x7f3a00074730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 --2- 172.21.15.64:0/2860306356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a00074730 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009900 tx=0x7f39fc02a480 comp rx=0 tx=0).stop 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 -- 172.21.15.64:0/2860306356 shutdown_connections 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 --2- 172.21.15.64:0/2860306356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0010e0f0 0x7f3a00110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 --2- 172.21.15.64:0/2860306356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a00074c70 0x7f3a0010dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 --2- 172.21.15.64:0/2860306356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a00074730 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 -- 172.21.15.64:0/2860306356 >> 172.21.15.64:0/2860306356 conn(0x7f3a000fd5f0 msgr2=0x7f3a000ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 -- 172.21.15.64:0/2860306356 shutdown_connections 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.199+0000 7f3a085fe640 1 -- 172.21.15.64:0/2860306356 wait complete. 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 Processor -- start 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 -- start start 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a0019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a00074c70 0x7f3a0019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0010e0f0 0x7f3a0019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3a00113ea0 con 0x7f3a00074350 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3a00113d20 con 0x7f3a0010e0f0 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a085fe640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a00114020 con 0x7f3a00074c70 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a0639a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a0019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a0639a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a0019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55512/0 (socket says 172.21.15.64:55512) 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a0639a640 1 -- 172.21.15.64:0/1523033955 learned_addr learned my addr 172.21.15.64:0/1523033955 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a05b99640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a00074c70 0x7f3a0019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.200+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0010e0f0 0x7f3a0019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a0639a640 1 -- 172.21.15.64:0/1523033955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a00074c70 msgr2=0x7f3a0019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a0639a640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a00074c70 0x7f3a0019a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a0639a640 1 -- 172.21.15.64:0/1523033955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0010e0f0 msgr2=0x7f3a0019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a0639a640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0010e0f0 0x7f3a0019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a0639a640 1 -- 172.21.15.64:0/1523033955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0019f380 con 0x7f3a00074350 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a0639a640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a0019a3d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f39fc006010 tx=0x7f39fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f39f37fe640 1 -- 172.21.15.64:0/1523033955 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc03f030 con 0x7f3a00074350 2024-08-19T01:45:15.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0019f610 con 0x7f3a00074350 2024-08-19T01:45:15.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f39f37fe640 1 -- 172.21.15.64:0/1523033955 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39fc003ce0 con 0x7f3a00074350 2024-08-19T01:45:15.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f39f37fe640 1 -- 172.21.15.64:0/1523033955 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc035070 con 0x7f3a00074350 2024-08-19T01:45:15.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.201+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a001a6f50 con 0x7f3a00074350 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.203+0000 7f39f37fe640 1 -- 172.21.15.64:0/1523033955 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f39fc042030 con 0x7f3a00074350 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 Processor -- start 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 -- start start 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5601b7651c60 con 0x5601b76eb990 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5601b75e9b90 con 0x5601b76ec2b0 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5601b764dc20 con 0x5601b76f6b70 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f39f37fe640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f39ec0a2cf0 0x7f39ec0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a06b9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a06b9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76ebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55526/0 (socket says 172.21.15.64:55526) 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.204+0000 7f3a06b9b640 1 -- 172.21.15.64:0/852508630 learned_addr learned my addr 172.21.15.64:0/852508630 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:45:15.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a05b99640 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a0639a640 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a06b9b640 1 -- 172.21.15.64:0/852508630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 msgr2=0x5601b76f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a06b9b640 1 -- 172.21.15.64:0/852508630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 msgr2=0x5601b76f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f39f37fe640 1 -- 172.21.15.64:0/1523033955 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f39fc0efa20 con 0x7f3a00074350 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a06b9b640 1 -- 172.21.15.64:0/852508630 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601b7361580 con 0x5601b76eb990 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a0639a640 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f39ec0a2cf0 0x7f39ec0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76ebd70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f39f8008e60 tx=0x7f39f8010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ceb7dca5d7261f2 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f39f0ff9640 1 -- 172.21.15.64:0/852508630 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8022070 con 0x5601b76eb990 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f39f0ff9640 1 -- 172.21.15.64:0/852508630 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f8011b70 con 0x5601b76eb990 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.205+0000 7f39f0ff9640 1 -- 172.21.15.64:0/852508630 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f801a400 con 0x5601b76eb990 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/852508630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 msgr2=0x5601b76ebd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f39ec0a2cf0 0x7f39ec0a51b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f39f801f000 tx=0x7f39f8020000 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76ebd70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f39f8008e60 tx=0x7f39f8010740 comp rx=0 tx=0).stop 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/852508630 shutdown_connections 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/852508630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76ebd70 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/852508630 >> 172.21.15.64:0/852508630 conn(0x5601b76e7f20 msgr2=0x5601b76e8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/852508630 shutdown_connections 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.206+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/852508630 wait complete. 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 -- start start 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 Processor -- start 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5601b76f9740 con 0x5601b76eb990 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5601b75ef7c0 con 0x5601b76ec2b0 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5601b76f98c0 con 0x5601b76f6b70 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0a9fc740 10 client.0 _ll_register_callbacks cb 0x5601b75f8f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0639a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0639a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55530/0 (socket says 172.21.15.64:55530) 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a0639a640 1 -- 172.21.15.64:0/1575260494 learned_addr learned my addr 172.21.15.64:0/1575260494 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.207+0000 7f3a05b99640 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0639a640 1 -- 172.21.15.64:0/1575260494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 msgr2=0x5601b76f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0639a640 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0639a640 1 -- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 msgr2=0x5601b76f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0639a640 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0639a640 1 -- 172.21.15.64:0/1575260494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601b742b960 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0639a640 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76f3a10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0b8050 tx=0x7f39fc042d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5e162764894cad server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f39e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f39e9ffb640 1 -- 172.21.15.64:0/1575260494 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc049040 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f39e9ffb640 1 -- 172.21.15.64:0/1575260494 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f39fc0b8260 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39f8011b70 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f39e9ffb640 1 -- 172.21.15.64:0/1575260494 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39fc03bcb0 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0a9fc740 10 client.15993 fetch_fsmap learned FSMap version 29 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5601b77056d0 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f39e9ffb640 1 -- 172.21.15.64:0/1575260494 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f8011b70 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f39e9ffb640 1 -- 172.21.15.64:0/1575260494 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f39fc049c40 con 0x5601b76eb990 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0a9fc740 10 client.15993 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0a9fc740 10 client.15993 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.208+0000 7f3a0a9fc740 20 client.15993 populate_metadata read hostname 'smithi064' 2024-08-19T01:45:15.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.209+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7706a70 0x5601b7708ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.209+0000 7f3a0a9fc740 4 client.15993 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:45:15.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.209+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601b765f1d0 con 0x5601b7706a70 2024-08-19T01:45:15.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.209+0000 7f3a05b99640 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7706a70 0x5601b7708ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.209+0000 7f3a05b99640 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7706a70 0x5601b7708ef0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=212f5751d15e9765 server_cookie=949d6cee0441aac8 in_seq=0 out_seq=0 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.209+0000 7f39e9ffb640 10 client.15993 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f39e9ffb640 1 -- 172.21.15.64:0/1575260494 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5601b765f1d0 con 0x5601b7706a70 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f39e9ffb640 10 client.15993 handle_command_reply: tid=0 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 client.15993 shutdown 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 msgr2=0x5601b76f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76f3a10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0b8050 tx=0x7f39fc042d70 comp rx=0 tx=0).stop 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 shutdown_connections 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7706a70 0x5601b7708ef0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/1575260494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76f3a10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 >> 172.21.15.64:0/1575260494 conn(0x5601b76e7f20 msgr2=0x5601b76e7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f39e9ffb640 0 client.15993 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:45:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 shutdown_connections 2024-08-19T01:45:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/1575260494 wait complete. 2024-08-19T01:45:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.210+0000 7f3a0a9fc740 20 client.15993 trim_cache size 0 max 16384 2024-08-19T01:45:15.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 Processor -- start 2024-08-19T01:45:15.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 -- start start 2024-08-19T01:45:15.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76eb990 0x5601b76f2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b770e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76f6b70 0x5601b770ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3a001b39d0 con 0x5601b76f6b70 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3a001b12a0 con 0x5601b76ec2b0 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a001af110 con 0x5601b76eb990 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76f6b70 0x5601b770ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76f6b70 0x5601b770ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55534/0 (socket says 172.21.15.64:55534) 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 -- 172.21.15.64:0/3337026346 learned_addr learned my addr 172.21.15.64:0/3337026346 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a0639a640 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76eb990 0x5601b76f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 -- 172.21.15.64:0/3337026346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76eb990 msgr2=0x5601b76f2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a05b99640 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b770e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76eb990 0x5601b76f2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 -- 172.21.15.64:0/3337026346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 msgr2=0x5601b770e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b770e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 -- 172.21.15.64:0/3337026346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601b76ee800 con 0x5601b76f6b70 2024-08-19T01:45:15.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.228+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76f6b70 0x5601b770ed00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f39f8010c40 tx=0x7f39f800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f6b9e418752901 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:15.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f39ebfff640 1 -- 172.21.15.64:0/3337026346 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8022030 con 0x5601b76f6b70 2024-08-19T01:45:15.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f39ebfff640 1 -- 172.21.15.64:0/3337026346 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f8011e50 con 0x5601b76f6b70 2024-08-19T01:45:15.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f39ebfff640 1 -- 172.21.15.64:0/3337026346 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f801c6d0 con 0x5601b76f6b70 2024-08-19T01:45:15.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3337026346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76f6b70 msgr2=0x5601b770ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76f6b70 0x5601b770ed00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f39f8010c40 tx=0x7f39f800f000 comp rx=0 tx=0).stop 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3337026346 shutdown_connections 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76f6b70 0x5601b770ed00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b770e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3337026346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76eb990 0x5601b76f2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3337026346 >> 172.21.15.64:0/3337026346 conn(0x5601b76e7f20 msgr2=0x5601b76f5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3337026346 shutdown_connections 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.229+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3337026346 wait complete. 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 -- start start 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 Processor -- start 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76fa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76fae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76fb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3a001af760 con 0x5601b76eb990 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3a001b0bd0 con 0x5601b76ec2b0 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a001afe30 con 0x5601b76f6b70 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0a9fc740 10 client.0 _ll_register_callbacks cb 0x7f39fc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76fa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76fa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55542/0 (socket says 172.21.15.64:55542) 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 -- 172.21.15.64:0/3932974108 learned_addr learned my addr 172.21.15.64:0/3932974108 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a06b9b640 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76fb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a05b99640 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76fae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 -- 172.21.15.64:0/3932974108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 msgr2=0x5601b76fb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76fb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 -- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 msgr2=0x5601b76fae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76fae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.230+0000 7f3a0639a640 1 -- 172.21.15.64:0/3932974108 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601b77707c0 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0639a640 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76fa8c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0c4330 tx=0x7f39fc009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b499a64ad485391f server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f39e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f39e9ffb640 1 -- 172.21.15.64:0/3932974108 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc0d8070 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f39e9ffb640 1 -- 172.21.15.64:0/3932974108 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f39fc0da2f0 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39f8011e50 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f39e9ffb640 1 -- 172.21.15.64:0/3932974108 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39fc0ad050 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 10 client.15999 fetch_fsmap learned FSMap version 29 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5601b77707c0 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f39e9ffb640 1 -- 172.21.15.64:0/3932974108 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f8011e50 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f39e9ffb640 1 -- 172.21.15.64:0/3932974108 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f39fc0bb040 con 0x5601b76eb990 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 10 client.15999 fetch_fsmap finished waiting for FSMap version 29 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 10 client.15999 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 20 client.15999 populate_metadata read hostname 'smithi064' 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7771e80 0x5601b7774270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 4 client.15999 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1a8e1bbc", "release": true, "await": true}] 2024-08-19T01:45:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.231+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1a8e1bbc", "release": true, "await": true}) -- 0x5601b76596e0 con 0x5601b7771e80 2024-08-19T01:45:15.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.232+0000 7f3a05b99640 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7771e80 0x5601b7774270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:45:15.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.232+0000 7f39e9ffb640 10 client.15999 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:45:15.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.232+0000 7f3a05b99640 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7771e80 0x5601b7774270 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4d912e63b6ddec4 server_cookie=5f916e298b6abec5 in_seq=0 out_seq=0 2024-08-19T01:45:15.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f39e9ffb640 1 -- 172.21.15.64:0/3932974108 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5601b76596e0 con 0x5601b7771e80 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f39e9ffb640 10 client.15999 handle_command_reply: tid=0 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 client.15999 shutdown 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 msgr2=0x5601b76fa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76fa8c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0c4330 tx=0x7f39fc009680 comp rx=0 tx=0).stop 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 shutdown_connections 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601b76f6b70 0x5601b76fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5601b7771e80 0x5601b7774270 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5601b76ec2b0 0x5601b76fae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 --2- 172.21.15.64:0/3932974108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601b76eb990 0x5601b76fa8c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 >> 172.21.15.64:0/3932974108 conn(0x5601b76e7f20 msgr2=0x5601b76ed430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f39e9ffb640 0 client.15999 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 shutdown_connections 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 1 -- 172.21.15.64:0/3932974108 wait complete. 2024-08-19T01:45:15.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.241+0000 7f3a0a9fc740 20 client.15999 trim_cache size 0 max 16384 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 29, 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 44, 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "1a8e1bbc": { 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 44, 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:45:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 140.2, 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:45:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:45:15.247 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:45:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f39ec0a2cf0 msgr2=0x7f39ec0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f39ec0a2cf0 0x7f39ec0a51b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f39f801f000 tx=0x7f39f8020000 comp rx=0 tx=0).stop 2024-08-19T01:45:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 msgr2=0x7f3a0019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:45:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a0019a3d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f39fc006010 tx=0x7f39fc0042f0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 shutdown_connections 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f39ec0a2cf0 0x7f39ec0a51b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0010e0f0 0x7f3a0019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a00074c70 0x7f3a0019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 --2- 172.21.15.64:0/1523033955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a00074350 0x7f3a0019a3d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 >> 172.21.15.64:0/1523033955 conn(0x7f3a000fd5f0 msgr2=0x7f3a000fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 shutdown_connections 2024-08-19T01:45:15.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:45:15.242+0000 7f3a085fe640 1 -- 172.21.15.64:0/1523033955 wait complete. 2024-08-19T01:45:15.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1017505 7.88 MB/sec execute 527 sec latency 19859.188 ms 2024-08-19T01:45:15.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:15 smithi064 ceph-mon[18515]: pgmap v375: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 9 op/s 2024-08-19T01:45:15.956 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1a8e1bbc', total seconds elapsed: 23.5 2024-08-19T01:45:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:15 smithi123 ceph-mon[21829]: pgmap v375: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 9 op/s 2024-08-19T01:45:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:15 smithi161 ceph-mon[21757]: pgmap v375: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 9 op/s 2024-08-19T01:45:16.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1018337 7.87 MB/sec execute 528 sec latency 72.271 ms 2024-08-19T01:45:17.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1019841 7.87 MB/sec execute 529 sec latency 52.682 ms 2024-08-19T01:45:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:17 smithi123 ceph-mon[21829]: pgmap v376: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 13 op/s 2024-08-19T01:45:18.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:17 smithi161 ceph-mon[21757]: pgmap v376: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 13 op/s 2024-08-19T01:45:18.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:17 smithi064 ceph-mon[18515]: pgmap v376: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 13 op/s 2024-08-19T01:45:18.411 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1021662 7.87 MB/sec execute 530 sec latency 27.069 ms 2024-08-19T01:45:19.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1023605 7.88 MB/sec execute 531 sec latency 24.828 ms 2024-08-19T01:45:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:19 smithi123 ceph-mon[21829]: pgmap v377: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-08-19T01:45:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:19 smithi161 ceph-mon[21757]: pgmap v377: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-08-19T01:45:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:19 smithi064 ceph-mon[18515]: pgmap v377: 129 pgs: 129 active+clean; 587 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-08-19T01:45:20.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1026039 7.89 MB/sec execute 532 sec latency 14.312 ms 2024-08-19T01:45:21.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1027479 7.89 MB/sec execute 533 sec latency 18.930 ms 2024-08-19T01:45:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:21 smithi123 ceph-mon[21829]: pgmap v378: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-08-19T01:45:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:21 smithi161 ceph-mon[21757]: pgmap v378: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-08-19T01:45:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:21 smithi064 ceph-mon[18515]: pgmap v378: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-08-19T01:45:22.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1030213 7.91 MB/sec execute 534 sec latency 19.000 ms 2024-08-19T01:45:23.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1031410 7.90 MB/sec execute 535 sec latency 18.279 ms 2024-08-19T01:45:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:23 smithi123 ceph-mon[21829]: pgmap v379: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-08-19T01:45:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:23 smithi161 ceph-mon[21757]: pgmap v379: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-08-19T01:45:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:23 smithi064 ceph-mon[18515]: pgmap v379: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-08-19T01:45:24.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1033807 7.91 MB/sec execute 536 sec latency 46.555 ms 2024-08-19T01:45:25.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1035308 7.91 MB/sec execute 537 sec latency 15.041 ms 2024-08-19T01:45:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:25 smithi123 ceph-mon[21829]: pgmap v380: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-08-19T01:45:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:25 smithi161 ceph-mon[21757]: pgmap v380: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-08-19T01:45:26.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:25 smithi064 ceph-mon[18515]: pgmap v380: 129 pgs: 129 active+clean; 546 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-08-19T01:45:26.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1037609 7.92 MB/sec execute 538 sec latency 24.863 ms 2024-08-19T01:45:27.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1039667 7.92 MB/sec execute 539 sec latency 24.420 ms 2024-08-19T01:45:28.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:27 smithi123 ceph-mon[21829]: pgmap v381: 129 pgs: 129 active+clean; 559 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-08-19T01:45:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:27 smithi161 ceph-mon[21757]: pgmap v381: 129 pgs: 129 active+clean; 559 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-08-19T01:45:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:27 smithi064 ceph-mon[18515]: pgmap v381: 129 pgs: 129 active+clean; 559 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-08-19T01:45:28.412 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1041288 7.93 MB/sec execute 540 sec latency 13.910 ms 2024-08-19T01:45:29.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1042769 7.92 MB/sec execute 541 sec latency 101.389 ms 2024-08-19T01:45:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: pgmap v382: 129 pgs: 129 active+clean; 559 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-08-19T01:45:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:45:30.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:29 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: pgmap v382: 129 pgs: 129 active+clean; 559 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:45:30.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:29 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: pgmap v382: 129 pgs: 129 active+clean; 559 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-08-19T01:45:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:45:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:45:30.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:45:30.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:29 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:45:30.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1044834 7.93 MB/sec execute 542 sec latency 14.673 ms 2024-08-19T01:45:31.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1046914 7.93 MB/sec execute 543 sec latency 15.793 ms 2024-08-19T01:45:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:31 smithi123 ceph-mon[21829]: pgmap v383: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 219 op/s 2024-08-19T01:45:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:31 smithi161 ceph-mon[21757]: pgmap v383: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 219 op/s 2024-08-19T01:45:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:31 smithi064 ceph-mon[18515]: pgmap v383: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.0 MiB/s wr, 219 op/s 2024-08-19T01:45:32.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1048688 7.94 MB/sec execute 544 sec latency 20.180 ms 2024-08-19T01:45:33.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1051312 7.95 MB/sec execute 545 sec latency 17.953 ms 2024-08-19T01:45:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:33 smithi123 ceph-mon[21829]: pgmap v384: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-08-19T01:45:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:33 smithi161 ceph-mon[21757]: pgmap v384: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-08-19T01:45:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:33 smithi064 ceph-mon[18515]: pgmap v384: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-08-19T01:45:34.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1052498 7.95 MB/sec execute 546 sec latency 14.213 ms 2024-08-19T01:45:35.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1055162 7.96 MB/sec execute 547 sec latency 12.249 ms 2024-08-19T01:45:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:35 smithi123 ceph-mon[21829]: pgmap v385: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-08-19T01:45:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:35 smithi161 ceph-mon[21757]: pgmap v385: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-08-19T01:45:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:35 smithi064 ceph-mon[18515]: pgmap v385: 129 pgs: 129 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 148 op/s 2024-08-19T01:45:36.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1056614 7.96 MB/sec execute 548 sec latency 33.778 ms 2024-08-19T01:45:37.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1058882 7.97 MB/sec execute 549 sec latency 16.088 ms 2024-08-19T01:45:38.036 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:37 smithi064 ceph-mon[18515]: pgmap v386: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-08-19T01:45:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:37 smithi123 ceph-mon[21829]: pgmap v386: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-08-19T01:45:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:37 smithi161 ceph-mon[21757]: pgmap v386: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.2 MiB/s wr, 214 op/s 2024-08-19T01:45:38.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1060473 7.97 MB/sec execute 550 sec latency 24.795 ms 2024-08-19T01:45:39.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1062387 7.98 MB/sec execute 551 sec latency 44.686 ms 2024-08-19T01:45:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:39 smithi123 ceph-mon[21829]: pgmap v387: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 141 op/s 2024-08-19T01:45:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:39 smithi161 ceph-mon[21757]: pgmap v387: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 141 op/s 2024-08-19T01:45:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:39 smithi064 ceph-mon[18515]: pgmap v387: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 141 op/s 2024-08-19T01:45:40.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1064131 7.97 MB/sec execute 552 sec latency 80.064 ms 2024-08-19T01:45:41.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1066143 7.98 MB/sec execute 553 sec latency 25.843 ms 2024-08-19T01:45:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:41 smithi123 ceph-mon[21829]: pgmap v388: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-08-19T01:45:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:41 smithi161 ceph-mon[21757]: pgmap v388: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-08-19T01:45:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:41 smithi064 ceph-mon[18515]: pgmap v388: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-08-19T01:45:42.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1068259 7.98 MB/sec execute 554 sec latency 13.785 ms 2024-08-19T01:45:43.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1069976 7.99 MB/sec execute 555 sec latency 55.811 ms 2024-08-19T01:45:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:43 smithi123 ceph-mon[21829]: pgmap v389: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-08-19T01:45:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:43 smithi161 ceph-mon[21757]: pgmap v389: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-08-19T01:45:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:43 smithi064 ceph-mon[18515]: pgmap v389: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-08-19T01:45:44.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1072755 8.01 MB/sec execute 556 sec latency 13.490 ms 2024-08-19T01:45:45.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1074013 8.00 MB/sec execute 557 sec latency 13.135 ms 2024-08-19T01:45:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:45 smithi123 ceph-mon[21829]: pgmap v390: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-08-19T01:45:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:45 smithi161 ceph-mon[21757]: pgmap v390: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-08-19T01:45:46.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:45 smithi064 ceph-mon[18515]: pgmap v390: 129 pgs: 129 active+clean; 600 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-08-19T01:45:46.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1076436 8.01 MB/sec execute 558 sec latency 12.866 ms 2024-08-19T01:45:47.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1077995 8.01 MB/sec execute 559 sec latency 41.894 ms 2024-08-19T01:45:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:47 smithi123 ceph-mon[21829]: pgmap v391: 129 pgs: 129 active+clean; 613 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:45:48.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:47 smithi161 ceph-mon[21757]: pgmap v391: 129 pgs: 129 active+clean; 613 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:45:48.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:47 smithi064 ceph-mon[18515]: pgmap v391: 129 pgs: 129 active+clean; 613 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 200 op/s 2024-08-19T01:45:48.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1080120 8.02 MB/sec execute 560 sec latency 24.559 ms 2024-08-19T01:45:49.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1081539 8.01 MB/sec execute 561 sec latency 85.233 ms 2024-08-19T01:45:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:49 smithi123 ceph-mon[21829]: pgmap v392: 129 pgs: 129 active+clean; 613 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 134 op/s 2024-08-19T01:45:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:49 smithi161 ceph-mon[21757]: pgmap v392: 129 pgs: 129 active+clean; 613 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 134 op/s 2024-08-19T01:45:50.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:49 smithi064 ceph-mon[18515]: pgmap v392: 129 pgs: 129 active+clean; 613 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 134 op/s 2024-08-19T01:45:50.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1083627 8.02 MB/sec execute 562 sec latency 72.480 ms 2024-08-19T01:45:51.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1085512 8.02 MB/sec execute 563 sec latency 19.063 ms 2024-08-19T01:45:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:51 smithi123 ceph-mon[21829]: pgmap v393: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 220 op/s 2024-08-19T01:45:52.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:51 smithi161 ceph-mon[21757]: pgmap v393: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 220 op/s 2024-08-19T01:45:52.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:51 smithi064 ceph-mon[18515]: pgmap v393: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 220 op/s 2024-08-19T01:45:52.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1087433 8.03 MB/sec execute 564 sec latency 16.799 ms 2024-08-19T01:45:53.413 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1089740 8.04 MB/sec execute 565 sec latency 14.037 ms 2024-08-19T01:45:54.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:53 smithi123 ceph-mon[21829]: pgmap v394: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 155 op/s 2024-08-19T01:45:54.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:53 smithi161 ceph-mon[21757]: pgmap v394: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 155 op/s 2024-08-19T01:45:54.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:53 smithi064 ceph-mon[18515]: pgmap v394: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 155 op/s 2024-08-19T01:45:54.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1091122 8.04 MB/sec execute 566 sec latency 40.839 ms 2024-08-19T01:45:55.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1092953 8.03 MB/sec execute 567 sec latency 75.786 ms 2024-08-19T01:45:56.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:55 smithi123 ceph-mon[21829]: pgmap v395: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 155 op/s 2024-08-19T01:45:56.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:55 smithi161 ceph-mon[21757]: pgmap v395: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 155 op/s 2024-08-19T01:45:56.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:55 smithi064 ceph-mon[18515]: pgmap v395: 129 pgs: 129 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 155 op/s 2024-08-19T01:45:56.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1094673 8.04 MB/sec execute 568 sec latency 14.732 ms 2024-08-19T01:45:57.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1097389 8.05 MB/sec execute 569 sec latency 15.485 ms 2024-08-19T01:45:58.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:57 smithi123 ceph-mon[21829]: pgmap v396: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.0 MiB/s wr, 232 op/s 2024-08-19T01:45:58.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:57 smithi161 ceph-mon[21757]: pgmap v396: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.0 MiB/s wr, 232 op/s 2024-08-19T01:45:58.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:57 smithi064 ceph-mon[18515]: pgmap v396: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.0 MiB/s wr, 232 op/s 2024-08-19T01:45:58.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1098581 8.05 MB/sec execute 570 sec latency 14.994 ms 2024-08-19T01:45:59.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1101218 8.06 MB/sec execute 571 sec latency 26.591 ms 2024-08-19T01:46:00.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:45:59 smithi123 ceph-mon[21829]: pgmap v397: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 163 op/s 2024-08-19T01:46:00.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:45:59 smithi161 ceph-mon[21757]: pgmap v397: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 163 op/s 2024-08-19T01:46:00.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:45:59 smithi064 ceph-mon[18515]: pgmap v397: 129 pgs: 129 active+clean; 586 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 163 op/s 2024-08-19T01:46:00.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1102544 8.06 MB/sec execute 572 sec latency 91.368 ms 2024-08-19T01:46:01.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1103431 8.05 MB/sec execute 573 sec latency 62.603 ms 2024-08-19T01:46:02.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:01 smithi123 ceph-mon[21829]: pgmap v398: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.6 MiB/s wr, 254 op/s 2024-08-19T01:46:02.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:01 smithi161 ceph-mon[21757]: pgmap v398: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.6 MiB/s wr, 254 op/s 2024-08-19T01:46:02.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:01 smithi064 ceph-mon[18515]: pgmap v398: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.6 MiB/s wr, 254 op/s 2024-08-19T01:46:02.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1104800 8.05 MB/sec execute 574 sec latency 67.984 ms 2024-08-19T01:46:03.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1106303 8.05 MB/sec execute 575 sec latency 19.002 ms 2024-08-19T01:46:04.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:03 smithi123 ceph-mon[21829]: pgmap v399: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 167 op/s 2024-08-19T01:46:04.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:03 smithi161 ceph-mon[21757]: pgmap v399: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 167 op/s 2024-08-19T01:46:04.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:03 smithi064 ceph-mon[18515]: pgmap v399: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 167 op/s 2024-08-19T01:46:04.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1108483 8.06 MB/sec execute 576 sec latency 19.470 ms 2024-08-19T01:46:05.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:04 smithi123 ceph-mon[21829]: pgmap v400: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 167 op/s 2024-08-19T01:46:05.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:04 smithi161 ceph-mon[21757]: pgmap v400: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 167 op/s 2024-08-19T01:46:05.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:04 smithi064 ceph-mon[18515]: pgmap v400: 129 pgs: 129 active+clean; 596 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 167 op/s 2024-08-19T01:46:05.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1109884 8.06 MB/sec execute 577 sec latency 52.252 ms 2024-08-19T01:46:06.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1111992 8.07 MB/sec execute 578 sec latency 31.153 ms 2024-08-19T01:46:07.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1113452 8.06 MB/sec execute 579 sec latency 21.470 ms 2024-08-19T01:46:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:07 smithi123 ceph-mon[21829]: pgmap v401: 129 pgs: 129 active+clean; 610 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 283 op/s 2024-08-19T01:46:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:07 smithi161 ceph-mon[21757]: pgmap v401: 129 pgs: 129 active+clean; 610 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 283 op/s 2024-08-19T01:46:07.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:07 smithi064 ceph-mon[18515]: pgmap v401: 129 pgs: 129 active+clean; 610 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 283 op/s 2024-08-19T01:46:08.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:08 smithi064 ceph-mon[18515]: osdmap e107: 12 total, 12 up, 12 in 2024-08-19T01:46:08.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1115590 8.07 MB/sec execute 580 sec latency 19.237 ms 2024-08-19T01:46:08.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:08 smithi123 ceph-mon[21829]: osdmap e107: 12 total, 12 up, 12 in 2024-08-19T01:46:08.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:08 smithi161 ceph-mon[21757]: osdmap e107: 12 total, 12 up, 12 in 2024-08-19T01:46:09.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1117220 8.07 MB/sec execute 581 sec latency 24.331 ms 2024-08-19T01:46:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:09 smithi123 ceph-mon[21829]: pgmap v403: 129 pgs: 129 active+clean; 610 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 247 op/s 2024-08-19T01:46:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:09 smithi161 ceph-mon[21757]: pgmap v403: 129 pgs: 129 active+clean; 610 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 247 op/s 2024-08-19T01:46:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:09 smithi064 ceph-mon[18515]: pgmap v403: 129 pgs: 129 active+clean; 610 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 247 op/s 2024-08-19T01:46:10.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1119131 8.08 MB/sec execute 582 sec latency 46.370 ms 2024-08-19T01:46:11.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1120853 8.07 MB/sec execute 583 sec latency 14.446 ms 2024-08-19T01:46:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:11 smithi123 ceph-mon[21829]: pgmap v404: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 245 op/s 2024-08-19T01:46:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:11 smithi161 ceph-mon[21757]: pgmap v404: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 245 op/s 2024-08-19T01:46:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:11 smithi064 ceph-mon[18515]: pgmap v404: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 245 op/s 2024-08-19T01:46:12.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1122920 8.08 MB/sec execute 584 sec latency 15.975 ms 2024-08-19T01:46:13.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1125012 8.08 MB/sec execute 585 sec latency 19.631 ms 2024-08-19T01:46:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:13 smithi123 ceph-mon[21829]: pgmap v405: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 245 op/s 2024-08-19T01:46:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:13 smithi161 ceph-mon[21757]: pgmap v405: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 245 op/s 2024-08-19T01:46:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:13 smithi064 ceph-mon[18515]: pgmap v405: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 245 op/s 2024-08-19T01:46:14.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1126648 8.09 MB/sec execute 586 sec latency 18.874 ms 2024-08-19T01:46:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:14 smithi123 ceph-mon[21829]: osdmap e108: 12 total, 12 up, 12 in 2024-08-19T01:46:15.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:14 smithi161 ceph-mon[21757]: osdmap e108: 12 total, 12 up, 12 in 2024-08-19T01:46:15.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:14 smithi064 ceph-mon[18515]: osdmap e108: 12 total, 12 up, 12 in 2024-08-19T01:46:15.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1128725 8.09 MB/sec execute 587 sec latency 123.594 ms 2024-08-19T01:46:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:15 smithi123 ceph-mon[21829]: pgmap v407: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 133 op/s 2024-08-19T01:46:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:15 smithi161 ceph-mon[21757]: pgmap v407: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 133 op/s 2024-08-19T01:46:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:15 smithi064 ceph-mon[18515]: pgmap v407: 129 pgs: 129 active+clean; 622 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 133 op/s 2024-08-19T01:46:16.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1130204 8.09 MB/sec execute 588 sec latency 20.631 ms 2024-08-19T01:46:17.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1133075 8.11 MB/sec execute 589 sec latency 15.055 ms 2024-08-19T01:46:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:17 smithi123 ceph-mon[21829]: pgmap v408: 129 pgs: 129 active+clean; 570 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.3 MiB/s wr, 241 op/s 2024-08-19T01:46:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:17 smithi161 ceph-mon[21757]: pgmap v408: 129 pgs: 129 active+clean; 570 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.3 MiB/s wr, 241 op/s 2024-08-19T01:46:18.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:17 smithi064 ceph-mon[18515]: pgmap v408: 129 pgs: 129 active+clean; 570 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.3 MiB/s wr, 241 op/s 2024-08-19T01:46:18.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1134566 8.11 MB/sec execute 590 sec latency 22.363 ms 2024-08-19T01:46:19.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1136773 8.11 MB/sec execute 591 sec latency 35.206 ms 2024-08-19T01:46:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:19 smithi123 ceph-mon[21829]: pgmap v409: 129 pgs: 129 active+clean; 570 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 215 op/s 2024-08-19T01:46:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:19 smithi161 ceph-mon[21757]: pgmap v409: 129 pgs: 129 active+clean; 570 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 215 op/s 2024-08-19T01:46:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:19 smithi064 ceph-mon[18515]: pgmap v409: 129 pgs: 129 active+clean; 570 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 215 op/s 2024-08-19T01:46:20.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1138109 8.11 MB/sec execute 592 sec latency 77.605 ms 2024-08-19T01:46:21.414 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1140307 8.12 MB/sec execute 593 sec latency 38.078 ms 2024-08-19T01:46:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:21 smithi123 ceph-mon[21829]: pgmap v410: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-08-19T01:46:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:21 smithi161 ceph-mon[21757]: pgmap v410: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-08-19T01:46:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:21 smithi064 ceph-mon[18515]: pgmap v410: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-08-19T01:46:22.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1141759 8.12 MB/sec execute 594 sec latency 16.283 ms 2024-08-19T01:46:23.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1143884 8.12 MB/sec execute 595 sec latency 13.508 ms 2024-08-19T01:46:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:23 smithi123 ceph-mon[21829]: pgmap v411: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-08-19T01:46:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:23 smithi161 ceph-mon[21757]: pgmap v411: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-08-19T01:46:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:23 smithi064 ceph-mon[18515]: pgmap v411: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-08-19T01:46:24.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1145570 8.12 MB/sec execute 596 sec latency 14.862 ms 2024-08-19T01:46:25.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1147560 8.13 MB/sec execute 597 sec latency 108.422 ms 2024-08-19T01:46:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:25 smithi123 ceph-mon[21829]: pgmap v412: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:46:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:25 smithi161 ceph-mon[21757]: pgmap v412: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:46:26.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:25 smithi064 ceph-mon[18515]: pgmap v412: 129 pgs: 129 active+clean; 575 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-08-19T01:46:26.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1149187 8.13 MB/sec execute 598 sec latency 22.727 ms 2024-08-19T01:46:27.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 1151262 8.14 MB/sec execute 599 sec latency 20.644 ms 2024-08-19T01:46:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:27 smithi123 ceph-mon[21829]: pgmap v413: 129 pgs: 129 active+clean; 586 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 269 op/s 2024-08-19T01:46:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:27 smithi161 ceph-mon[21757]: pgmap v413: 129 pgs: 129 active+clean; 586 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 269 op/s 2024-08-19T01:46:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:27 smithi064 ceph-mon[18515]: pgmap v413: 129 pgs: 129 active+clean; 586 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 269 op/s 2024-08-19T01:46:28.415 INFO:tasks.workunit.client.0.smithi064.stdout: 1 cleanup 600 sec 2024-08-19T01:46:28.613 INFO:tasks.workunit.client.0.smithi064.stdout: 0 cleanup 600 sec 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Operation Count AvgLat MaxLat 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: ---------------------------------------- 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: NTCreateX 155442 1.458 23159.019 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Close 114187 0.002 0.142 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Rename 6587 8.868 30169.985 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Unlink 31390 2.130 20715.408 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Deltree 4 78.112 89.620 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Mkdir 2 0.004 0.004 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Qpathinfo 140893 0.865 166.988 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Qfileinfo 24701 0.001 6.229 2024-08-19T01:46:28.614 INFO:tasks.workunit.client.0.smithi064.stdout: Qfsinfo 25841 0.148 4.927 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: Sfileinfo 12672 4.241 29694.720 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: Find 54481 0.120 113.337 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: WriteX 77526 0.365 19277.282 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: ReadX 243752 0.026 34.413 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: LockX 506 0.704 1.816 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: UnlockX 506 0.650 4.251 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: Flush 10895 4.088 223.417 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout: 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stdout:Throughput 8.13571 MB/sec 1 clients 1 procs max_latency=30170.332 ms 2024-08-19T01:46:28.615 INFO:tasks.workunit.client.0.smithi064.stderr:+ dbench 10 2024-08-19T01:46:28.616 INFO:tasks.workunit.client.0.smithi064.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-08-19T01:46:28.616 INFO:tasks.workunit.client.0.smithi064.stdout: 2024-08-19T01:46:28.616 INFO:tasks.workunit.client.0.smithi064.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-08-19T01:46:28.616 INFO:tasks.workunit.client.0.smithi064.stdout:8 of 10 processes prepared for launch 0 sec 2024-08-19T01:46:28.715 INFO:tasks.workunit.client.0.smithi064.stdout:10 of 10 processes prepared for launch 0 sec 2024-08-19T01:46:28.715 INFO:tasks.workunit.client.0.smithi064.stdout:releasing clients 2024-08-19T01:46:29.715 INFO:tasks.workunit.client.0.smithi064.stdout: 10 435 153.33 MB/sec warmup 1 sec latency 46.285 ms 2024-08-19T01:46:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:29 smithi123 ceph-mon[21829]: pgmap v414: 129 pgs: 129 active+clean; 586 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-08-19T01:46:30.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:29 smithi161 ceph-mon[21757]: pgmap v414: 129 pgs: 129 active+clean; 586 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-08-19T01:46:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:29 smithi064 ceph-mon[18515]: pgmap v414: 129 pgs: 129 active+clean; 586 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-08-19T01:46:30.715 INFO:tasks.workunit.client.0.smithi064.stdout: 10 842 107.76 MB/sec warmup 2 sec latency 194.702 ms 2024-08-19T01:46:30.800 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.1 2024-08-19T01:46:30.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:46:31.104 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:46:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:30 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:46:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:30 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:46:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:30 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:46:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:30 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:46:31.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:30 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:46:31.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:30 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:46:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:30 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:46:31.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:30 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:46:31.228 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:30 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:46:31.228 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:30 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:46:31.228 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:30 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:46:31.228 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:30 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:46:31.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 -- 172.21.15.64:0/2426455792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4100b50 msgr2=0x7f20d4100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 --2- 172.21.15.64:0/2426455792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4100b50 0x7f20d4100f30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009900 tx=0x7f20c402a480 comp rx=0 tx=0).stop 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 -- 172.21.15.64:0/2426455792 shutdown_connections 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 --2- 172.21.15.64:0/2426455792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4102c20 0x7f20d410d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 --2- 172.21.15.64:0/2426455792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d41022a0 0x7f20d41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 --2- 172.21.15.64:0/2426455792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4100b50 0x7f20d4100f30 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 -- 172.21.15.64:0/2426455792 >> 172.21.15.64:0/2426455792 conn(0x7f20d40fc500 msgr2=0x7f20d40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 -- 172.21.15.64:0/2426455792 shutdown_connections 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.606+0000 7f20d900f640 1 -- 172.21.15.64:0/2426455792 wait complete. 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.607+0000 7f20d900f640 1 Processor -- start 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.607+0000 7f20d900f640 1 -- start start 2024-08-19T01:46:31.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d900f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4100b50 0x7f20d4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d900f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d41022a0 0x7f20d4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d900f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4102c20 0x7f20d419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d900f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4110cb0 con 0x7f20d4102c20 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d900f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4110b30 con 0x7f20d41022a0 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d900f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20d4110e30 con 0x7f20d4100b50 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4100b50 0x7f20d4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d359e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4102c20 0x7f20d419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4100b50 0x7f20d4198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:54520/0 (socket says 172.21.15.64:54520) 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 -- 172.21.15.64:0/2517032719 learned_addr learned my addr 172.21.15.64:0/2517032719 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d259c640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d41022a0 0x7f20d4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 -- 172.21.15.64:0/2517032719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d41022a0 msgr2=0x7f20d4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d41022a0 0x7f20d4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 -- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4102c20 msgr2=0x7f20d419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4102c20 0x7f20d419cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 -- 172.21.15.64:0/2517032719 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d419d1e0 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20d2d9d640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4100b50 0x7f20d4198230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f20c4006010 tx=0x7f20c4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.608+0000 7f20bbfff640 1 -- 172.21.15.64:0/2517032719 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c403f070 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.609+0000 7f20bbfff640 1 -- 172.21.15.64:0/2517032719 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20c4031c30 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.609+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d419d470 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.609+0000 7f20bbfff640 1 -- 172.21.15.64:0/2517032719 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4035030 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.609+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d40fc8e0 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.610+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d4101ee0 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.611+0000 7f20bbfff640 1 -- 172.21.15.64:0/2517032719 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f20c4042070 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.611+0000 7f20bbfff640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f20bc0a2cf0 0x7f20bc0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.612+0000 7f20d259c640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f20bc0a2cf0 0x7f20bc0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.612+0000 7f20bbfff640 1 -- 172.21.15.64:0/2517032719 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f20c40efee0 con 0x7f20d4100b50 2024-08-19T01:46:31.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.612+0000 7f20d259c640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f20bc0a2cf0 0x7f20bc0a51b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f20d40ffc80 tx=0x7f20c8005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:31.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.616+0000 7f20bbfff640 1 -- 172.21.15.64:0/2517032719 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f20c4031de0 con 0x7f20d4100b50 2024-08-19T01:46:31.715 INFO:tasks.workunit.client.0.smithi064.stdout: 10 1491 86.87 MB/sec warmup 3 sec latency 40.077 ms 2024-08-19T01:46:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.858+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20d4100f30 con 0x7f20d4100b50 2024-08-19T01:46:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.859+0000 7f20bbfff640 1 -- 172.21.15.64:0/2517032719 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11924 (secure 0 0 0) 0x7f20c40bc080 con 0x7f20d4100b50 2024-08-19T01:46:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:46:31.860 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"btime":"2024-08-19T01:46:31:794441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:46:31.560051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:46:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-08-19T01:46:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f20bc0a2cf0 msgr2=0x7f20bc0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f20bc0a2cf0 0x7f20bc0a51b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f20d40ffc80 tx=0x7f20c8005f90 comp rx=0 tx=0).stop 2024-08-19T01:46:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4100b50 msgr2=0x7f20d4198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4100b50 0x7f20d4198230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f20c4006010 tx=0x7f20c4003c10 comp rx=0 tx=0).stop 2024-08-19T01:46:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 shutdown_connections 2024-08-19T01:46:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f20bc0a2cf0 0x7f20bc0a51b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20d4102c20 0x7f20d419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d41022a0 0x7f20d4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 --2- 172.21.15.64:0/2517032719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20d4100b50 0x7f20d4198230 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 >> 172.21.15.64:0/2517032719 conn(0x7f20d40fc500 msgr2=0x7f20d4069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 shutdown_connections 2024-08-19T01:46:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:31.863+0000 7f20d900f640 1 -- 172.21.15.64:0/2517032719 wait complete. 2024-08-19T01:46:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:31 smithi123 ceph-mon[21829]: pgmap v415: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.3 MiB/s wr, 258 op/s 2024-08-19T01:46:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:31 smithi161 ceph-mon[21757]: pgmap v415: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.3 MiB/s wr, 258 op/s 2024-08-19T01:46:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:31 smithi064 ceph-mon[18515]: pgmap v415: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.3 MiB/s wr, 258 op/s 2024-08-19T01:46:32.715 INFO:tasks.workunit.client.0.smithi064.stdout: 10 2615 74.63 MB/sec warmup 4 sec latency 97.591 ms 2024-08-19T01:46:32.842 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 145.1 --await-for 45.0' 2024-08-19T01:46:32.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 145.1 --await-for 45.0 2024-08-19T01:46:33.131 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:46:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:32 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:46:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:32 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2517032719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:46:33.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:32 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:46:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:32 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2517032719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:46:33.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:32 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:46:33.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:32 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2517032719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:46:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 -- 172.21.15.64:0/2539743900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc102ad0 msgr2=0x7f5adc102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/2539743900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc102ad0 0x7f5adc102eb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009900 tx=0x7f5ad002a480 comp rx=0 tx=0).stop 2024-08-19T01:46:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 -- 172.21.15.64:0/2539743900 shutdown_connections 2024-08-19T01:46:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/2539743900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc10beb0 0x7f5adc10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/2539743900 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc1033f0 0x7f5adc10b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/2539743900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc102ad0 0x7f5adc102eb0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 -- 172.21.15.64:0/2539743900 >> 172.21.15.64:0/2539743900 conn(0x7f5adc0fc4d0 msgr2=0x7f5adc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 -- 172.21.15.64:0/2539743900 shutdown_connections 2024-08-19T01:46:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.654+0000 7f5ae2d97640 1 -- 172.21.15.64:0/2539743900 wait complete. 2024-08-19T01:46:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 Processor -- start 2024-08-19T01:46:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 -- start start 2024-08-19T01:46:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc100730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc1011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111c90 con 0x7f5adc10beb0 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111b10 con 0x7f5adc1033f0 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.655+0000 7f5ae2d97640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111e10 con 0x7f5adc102ad0 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc100730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc100730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:54536/0 (socket says 172.21.15.64:54536) 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3260503842 learned_addr learned my addr 172.21.15.64:0/3260503842 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae1334640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc1011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3260503842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 msgr2=0x7f5adc100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 msgr2=0x7f5adc1011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc1011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3260503842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc1a66b0 con 0x7f5adc102ad0 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5adbfff640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae1334640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.656+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc100730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006010 tx=0x7f5ad0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.657+0000 7f5ad9ffb640 1 -- 172.21.15.64:0/3260503842 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad003f030 con 0x7f5adc102ad0 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.657+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc1a6940 con 0x7f5adc102ad0 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.657+0000 7f5ad9ffb640 1 -- 172.21.15.64:0/3260503842 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ad0004440 con 0x7f5adc102ad0 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.657+0000 7f5ad9ffb640 1 -- 172.21.15.64:0/3260503842 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0035070 con 0x7f5adc102ad0 2024-08-19T01:46:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.657+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc1a6e70 con 0x7f5adc102ad0 2024-08-19T01:46:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.659+0000 7f5ad9ffb640 1 -- 172.21.15.64:0/3260503842 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5ad0042030 con 0x7f5adc102ad0 2024-08-19T01:46:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.660+0000 7f5ae5195740 1 Processor -- start 2024-08-19T01:46:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.660+0000 7f5ae5195740 1 -- start start 2024-08-19T01:46:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b98430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71b98970 0x55ef71bab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71babb20 0x55ef71badf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae5195740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ef71afda70 con 0x55ef71b98050 2024-08-19T01:46:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae5195740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ef71a95780 con 0x55ef71babb20 2024-08-19T01:46:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae5195740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ef71af9a30 con 0x55ef71b98970 2024-08-19T01:46:33.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ad9ffb640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5acc0a2cf0 0x7f5acc0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b98430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b98430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60862/0 (socket says 172.21.15.64:60862) 2024-08-19T01:46:33.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 -- 172.21.15.64:0/1147173961 learned_addr learned my addr 172.21.15.64:0/1147173961 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:33.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71b98970 0x55ef71bab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5adbfff640 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71babb20 0x55ef71badf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5acc0a2cf0 0x7f5acc0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ad9ffb640 1 -- 172.21.15.64:0/3260503842 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f5ad00efba0 con 0x7f5adc102ad0 2024-08-19T01:46:33.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 -- 172.21.15.64:0/1147173961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71b98970 msgr2=0x55ef71bab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71b98970 0x55ef71bab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 -- 172.21.15.64:0/1147173961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71babb20 msgr2=0x55ef71badf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71babb20 0x55ef71badf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.661+0000 7f5ae1334640 1 -- 172.21.15.64:0/1147173961 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef7186fb90 con 0x55ef71b98050 2024-08-19T01:46:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.662+0000 7f5ae1334640 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b98430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4011cc0 tx=0x7f5ad400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66787149c0cb0d19 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.662+0000 7f5ac6ffd640 1 -- 172.21.15.64:0/1147173961 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad401ece0 con 0x55ef71b98050 2024-08-19T01:46:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.662+0000 7f5ae1334640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5acc0a2cf0 0x7f5acc0a51b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40099e0 tx=0x7f5ad401f000 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.662+0000 7f5ac6ffd640 1 -- 172.21.15.64:0/1147173961 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ad4022070 con 0x55ef71b98050 2024-08-19T01:46:33.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.662+0000 7f5ac6ffd640 1 -- 172.21.15.64:0/1147173961 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad401d640 con 0x55ef71b98050 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 -- 172.21.15.64:0/1147173961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 msgr2=0x55ef71b98430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b98430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4011cc0 tx=0x7f5ad400f790 comp rx=0 tx=0).stop 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 -- 172.21.15.64:0/1147173961 shutdown_connections 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71babb20 0x55ef71badf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71b98970 0x55ef71bab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 --2- 172.21.15.64:0/1147173961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b98430 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 -- 172.21.15.64:0/1147173961 >> 172.21.15.64:0/1147173961 conn(0x55ef71b946c0 msgr2=0x55ef71b94aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:33.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 -- 172.21.15.64:0/1147173961 shutdown_connections 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.663+0000 7f5ae5195740 1 -- 172.21.15.64:0/1147173961 wait complete. 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 -- start start 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 Processor -- start 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98050 0x55ef71ba8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98970 0x55ef71ba8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71ba94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ef71bae6d0 con 0x55ef71b98970 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ef71ab8990 con 0x55ef71b98050 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5ae5195740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ef71bae850 con 0x55ef71babb20 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.664+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98970 0x55ef71ba8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5ae0b33640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98050 0x55ef71ba8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5ae5195740 10 client.0 _ll_register_callbacks cb 0x55ef7180d090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:46:33.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98970 0x55ef71ba8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60868/0 (socket says 172.21.15.64:60868) 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 -- 172.21.15.64:0/3381007347 learned_addr learned my addr 172.21.15.64:0/3381007347 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5ae1334640 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71ba94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 -- 172.21.15.64:0/3381007347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 msgr2=0x55ef71ba94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71ba94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 -- 172.21.15.64:0/3381007347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98050 msgr2=0x55ef71ba8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98050 0x55ef71ba8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 -- 172.21.15.64:0/3381007347 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef71ab76c0 con 0x55ef71b98970 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5adbfff640 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98970 0x55ef71ba8fa0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8012d50 tx=0x7f5ac800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb739fd8da743cd server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5abbfff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5abbfff640 1 -- 172.21.15.64:0/3381007347 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8012110 con 0x55ef71b98970 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.665+0000 7f5abbfff640 1 -- 172.21.15.64:0/3381007347 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f5ac8004050 con 0x55ef71b98970 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ad4022070 con 0x55ef71b98970 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5abbfff640 1 -- 172.21.15.64:0/3381007347 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ac8025050 con 0x55ef71b98970 2024-08-19T01:46:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 10 client.16017 fetch_fsmap learned FSMap version 30 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ef71bba450 con 0x55ef71b98970 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5abbfff640 1 -- 172.21.15.64:0/3381007347 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ad4022070 con 0x55ef71b98970 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5abbfff640 1 -- 172.21.15.64:0/3381007347 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5ac8010430 con 0x55ef71b98970 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 10 client.16017 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 10 client.16017 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 20 client.16017 populate_metadata read hostname 'smithi064' 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71bbb910 0x55ef71bbdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 4 client.16017 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.666+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef71b0b1a0 con 0x55ef71bbb910 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.667+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71bbb910 0x55ef71bbdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.667+0000 7f5abbfff640 10 client.16017 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.667+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71bbb910 0x55ef71bbdd80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eedd6c76da86068d server_cookie=41f628e40815c82a in_seq=0 out_seq=0 2024-08-19T01:46:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.672+0000 7f5abbfff640 1 -- 172.21.15.64:0/3381007347 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ef71b0b1a0 con 0x55ef71bbb910 2024-08-19T01:46:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.672+0000 7f5abbfff640 10 client.16017 handle_command_reply: tid=0 2024-08-19T01:46:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.672+0000 7f5ae5195740 1 client.16017 shutdown 2024-08-19T01:46:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98970 msgr2=0x55ef71ba8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98970 0x55ef71ba8fa0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8012d50 tx=0x7f5ac800f790 comp rx=0 tx=0).stop 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 shutdown_connections 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71ba94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98970 0x55ef71ba8fa0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71bbb910 0x55ef71bbdd80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3381007347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98050 0x55ef71ba8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5abbfff640 0 client.16017 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 >> 172.21.15.64:0/3381007347 conn(0x55ef71b946c0 msgr2=0x55ef71b9a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:33.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 shutdown_connections 2024-08-19T01:46:33.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 1 -- 172.21.15.64:0/3381007347 wait complete. 2024-08-19T01:46:33.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.673+0000 7f5ae5195740 20 client.16017 trim_cache size 0 max 16384 2024-08-19T01:46:33.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 Processor -- start 2024-08-19T01:46:33.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 -- start start 2024-08-19T01:46:33.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b93600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bc3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bc3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1af5b0 con 0x55ef71b98050 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1ace80 con 0x55ef71b98970 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae5195740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1aad10 con 0x55ef71babb20 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae0b33640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b93600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae0b33640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b93600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60882/0 (socket says 172.21.15.64:60882) 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3712713476 learned_addr learned my addr 172.21.15.64:0/3712713476 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5adbfff640 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bc3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.694+0000 7f5ae1334640 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bc3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3712713476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 msgr2=0x55ef71bc3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bc3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3712713476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 msgr2=0x55ef71bc3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bc3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ae0b33640 1 -- 172.21.15.64:0/3712713476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef71b93f70 con 0x55ef71b98050 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5adbfff640 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bc3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b93600 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00b81c0 tx=0x7f5ad00cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89700a0c0fbae23 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ac67fc640 1 -- 172.21.15.64:0/3712713476 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0049040 con 0x55ef71b98050 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ac67fc640 1 -- 172.21.15.64:0/3712713476 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ad00c4920 con 0x55ef71b98050 2024-08-19T01:46:33.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.695+0000 7f5ac67fc640 1 -- 172.21.15.64:0/3712713476 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad00cd880 con 0x55ef71b98050 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 -- 172.21.15.64:0/3712713476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 msgr2=0x55ef71b93600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b93600 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00b81c0 tx=0x7f5ad00cf040 comp rx=0 tx=0).stop 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 -- 172.21.15.64:0/3712713476 shutdown_connections 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bc3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bc3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 --2- 172.21.15.64:0/3712713476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71b93600 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 -- 172.21.15.64:0/3712713476 >> 172.21.15.64:0/3712713476 conn(0x55ef71b946c0 msgr2=0x55ef71b92ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:33.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 -- 172.21.15.64:0/3712713476 shutdown_connections 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 -- 172.21.15.64:0/3712713476 wait complete. 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 -- start start 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.696+0000 7f5ae5195740 1 Processor -- start 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71bb07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bb0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae5195740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bb1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae5195740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1ab360 con 0x55ef71b98050 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae5195740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1ac7b0 con 0x55ef71b98970 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae5195740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1aba10 con 0x55ef71babb20 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae5195740 10 client.0 _ll_register_callbacks cb 0x7f5ac8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bb0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bb0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:58598/0 (socket says 172.21.15.64:58598) 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 -- 172.21.15.64:0/4116171962 learned_addr learned my addr 172.21.15.64:0/4116171962 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:33.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae1334640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bb1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5ae0b33640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71bb07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 -- 172.21.15.64:0/4116171962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 msgr2=0x55ef71bb1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bb1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 -- 172.21.15.64:0/4116171962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 msgr2=0x55ef71bb07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71bb07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.697+0000 7f5adbfff640 1 -- 172.21.15.64:0/4116171962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef71b92c70 con 0x55ef71b98970 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5adbfff640 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bb0d30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8010800 tx=0x7f5ac800fd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6caf3da11979cf server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5abbfff640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-08-19T01:46:33.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5abbfff640 1 -- 172.21.15.64:0/4116171962 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8022040 con 0x55ef71b98970 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ad00c4920 con 0x55ef71b98970 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5abbfff640 1 -- 172.21.15.64:0/4116171962 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f5ac8010430 con 0x55ef71b98970 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5abbfff640 1 -- 172.21.15.64:0/4116171962 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ac803a050 con 0x55ef71b98970 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5ae5195740 10 client.25442 fetch_fsmap learned FSMap version 30 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ef71c27730 con 0x55ef71b98970 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.698+0000 7f5abbfff640 1 -- 172.21.15.64:0/4116171962 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ad00c4920 con 0x55ef71b98970 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5abbfff640 1 -- 172.21.15.64:0/4116171962 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5ac80035f0 con 0x55ef71b98970 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae5195740 10 client.25442 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae5195740 10 client.25442 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae5195740 20 client.25442 populate_metadata read hostname 'smithi064' 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae5195740 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71c27fb0 0x55ef71c2a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae5195740 4 client.25442 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.1, "await_for": 45.0}] 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.1, "await_for": 45.0}) -- 0x55ef71b054f0 con 0x55ef71c27fb0 2024-08-19T01:46:33.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71c27fb0 0x55ef71c2a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:33.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5abbfff640 10 client.25442 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:33.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:33.699+0000 7f5ae0b33640 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71c27fb0 0x55ef71c2a3d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5652365c85966b61 server_cookie=56793529054148b in_seq=0 out_seq=0 2024-08-19T01:46:33.715 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3393 71.67 MB/sec warmup 5 sec latency 50.043 ms 2024-08-19T01:46:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:33 smithi123 ceph-mon[21829]: pgmap v416: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-08-19T01:46:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:33 smithi161 ceph-mon[21757]: pgmap v416: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-08-19T01:46:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:33 smithi064 ceph-mon[18515]: pgmap v416: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5abbfff640 1 -- 172.21.15.64:0/4116171962 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ef71b054f0 con 0x55ef71c27fb0 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5abbfff640 10 client.25442 handle_command_reply: tid=0 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 client.25442 shutdown 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 msgr2=0x55ef71bb0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bb0d30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8010800 tx=0x7f5ac800fd00 comp rx=0 tx=0).stop 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 shutdown_connections 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ef71babb20 0x55ef71bb1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ef71c27fb0 0x55ef71c2a3d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ef71b98970 0x55ef71bb0d30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 --2- 172.21.15.64:0/4116171962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ef71b98050 0x55ef71bb07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5abbfff640 0 client.25442 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 >> 172.21.15.64:0/4116171962 conn(0x55ef71b946c0 msgr2=0x55ef71ba9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 shutdown_connections 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.545+0000 7f5ae5195740 1 -- 172.21.15.64:0/4116171962 wait complete. 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae5195740 20 client.25442 trim_cache size 0 max 16384 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 46, 2024-08-19T01:46:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "9bec36f3": { 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 46, 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 145.1, 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:46:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:46:34.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5acc0a2cf0 msgr2=0x7f5acc0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5acc0a2cf0 0x7f5acc0a51b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40099e0 tx=0x7f5ad401f000 comp rx=0 tx=0).stop 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 msgr2=0x7f5adc100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc100730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006010 tx=0x7f5ad0004040 comp rx=0 tx=0).stop 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 shutdown_connections 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5acc0a2cf0 0x7f5acc0a51b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5adc10beb0 0x7f5adc1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5adc1033f0 0x7f5adc100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 --2- 172.21.15.64:0/3260503842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5adc102ad0 0x7f5adc100730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.546+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 >> 172.21.15.64:0/3260503842 conn(0x7f5adc0fc4d0 msgr2=0x7f5adc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.547+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 shutdown_connections 2024-08-19T01:46:34.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:34.547+0000 7f5ae2d97640 1 -- 172.21.15.64:0/3260503842 wait complete. 2024-08-19T01:46:34.715 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 59.78 MB/sec warmup 6 sec latency 996.961 ms 2024-08-19T01:46:35.299 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9bec36f3', quiesce took 4.5 seconds. Will release after: 20.6 2024-08-19T01:46:35.715 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 51.24 MB/sec warmup 7 sec latency 1997.024 ms 2024-08-19T01:46:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:35 smithi123 ceph-mon[21829]: pgmap v417: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-08-19T01:46:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:35 smithi161 ceph-mon[21757]: pgmap v417: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-08-19T01:46:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:35 smithi064 ceph-mon[18515]: pgmap v417: 129 pgs: 129 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-08-19T01:46:36.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 44.84 MB/sec warmup 8 sec latency 2997.126 ms 2024-08-19T01:46:37.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 39.85 MB/sec warmup 9 sec latency 3997.225 ms 2024-08-19T01:46:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:37 smithi123 ceph-mon[21829]: pgmap v418: 129 pgs: 129 active+clean; 625 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.8 MiB/s wr, 382 op/s 2024-08-19T01:46:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:37 smithi161 ceph-mon[21757]: pgmap v418: 129 pgs: 129 active+clean; 625 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.8 MiB/s wr, 382 op/s 2024-08-19T01:46:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:37 smithi064 ceph-mon[18515]: pgmap v418: 129 pgs: 129 active+clean; 625 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.8 MiB/s wr, 382 op/s 2024-08-19T01:46:38.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 35.87 MB/sec warmup 10 sec latency 4997.298 ms 2024-08-19T01:46:39.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 32.61 MB/sec warmup 11 sec latency 5997.405 ms 2024-08-19T01:46:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:39 smithi123 ceph-mon[21829]: pgmap v419: 129 pgs: 129 active+clean; 625 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 288 op/s 2024-08-19T01:46:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:39 smithi161 ceph-mon[21757]: pgmap v419: 129 pgs: 129 active+clean; 625 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 288 op/s 2024-08-19T01:46:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:39 smithi064 ceph-mon[18515]: pgmap v419: 129 pgs: 129 active+clean; 625 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 288 op/s 2024-08-19T01:46:40.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 29.89 MB/sec warmup 12 sec latency 6997.483 ms 2024-08-19T01:46:41.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 27.59 MB/sec warmup 13 sec latency 7997.559 ms 2024-08-19T01:46:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:41 smithi123 ceph-mon[21829]: pgmap v420: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-08-19T01:46:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:41 smithi161 ceph-mon[21757]: pgmap v420: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-08-19T01:46:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:41 smithi064 ceph-mon[18515]: pgmap v420: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-08-19T01:46:42.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 25.62 MB/sec warmup 14 sec latency 8997.618 ms 2024-08-19T01:46:43.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:42 smithi123 ceph-mon[21829]: pgmap v421: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 237 op/s 2024-08-19T01:46:43.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:42 smithi161 ceph-mon[21757]: pgmap v421: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 237 op/s 2024-08-19T01:46:43.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:42 smithi064 ceph-mon[18515]: pgmap v421: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 237 op/s 2024-08-19T01:46:43.716 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 23.91 MB/sec warmup 15 sec latency 9997.705 ms 2024-08-19T01:46:44.717 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 22.42 MB/sec warmup 16 sec latency 10997.813 ms 2024-08-19T01:46:45.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:45 smithi123 ceph-mon[21829]: pgmap v422: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 237 op/s 2024-08-19T01:46:45.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:45 smithi161 ceph-mon[21757]: pgmap v422: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 237 op/s 2024-08-19T01:46:45.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:45 smithi064 ceph-mon[18515]: pgmap v422: 129 pgs: 129 active+clean; 624 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 237 op/s 2024-08-19T01:46:45.717 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 21.10 MB/sec warmup 17 sec latency 11997.917 ms 2024-08-19T01:46:46.717 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 19.93 MB/sec warmup 18 sec latency 12998.015 ms 2024-08-19T01:46:47.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:47 smithi123 ceph-mon[21829]: pgmap v423: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-08-19T01:46:47.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:47 smithi161 ceph-mon[21757]: pgmap v423: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-08-19T01:46:47.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:47 smithi064 ceph-mon[18515]: pgmap v423: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-08-19T01:46:47.717 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 18.88 MB/sec warmup 19 sec latency 13998.117 ms 2024-08-19T01:46:48.717 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 17.93 MB/sec warmup 20 sec latency 14998.210 ms 2024-08-19T01:46:49.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:49 smithi123 ceph-mon[21829]: pgmap v424: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.4 MiB/s wr, 59 op/s 2024-08-19T01:46:49.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:49 smithi161 ceph-mon[21757]: pgmap v424: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.4 MiB/s wr, 59 op/s 2024-08-19T01:46:49.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:49 smithi064 ceph-mon[18515]: pgmap v424: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.4 MiB/s wr, 59 op/s 2024-08-19T01:46:49.717 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 17.08 MB/sec warmup 21 sec latency 15998.255 ms 2024-08-19T01:46:50.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 16.30 MB/sec warmup 22 sec latency 16999.144 ms 2024-08-19T01:46:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:51 smithi123 ceph-mon[21829]: pgmap v425: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-08-19T01:46:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:51 smithi161 ceph-mon[21757]: pgmap v425: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-08-19T01:46:51.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:51 smithi064 ceph-mon[18515]: pgmap v425: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-08-19T01:46:51.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 15.59 MB/sec warmup 23 sec latency 17999.194 ms 2024-08-19T01:46:52.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 14.94 MB/sec warmup 24 sec latency 18999.307 ms 2024-08-19T01:46:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:53 smithi123 ceph-mon[21829]: pgmap v426: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 58 op/s 2024-08-19T01:46:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:53 smithi161 ceph-mon[21757]: pgmap v426: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 58 op/s 2024-08-19T01:46:53.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:53 smithi064 ceph-mon[18515]: pgmap v426: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 58 op/s 2024-08-19T01:46:53.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 14.35 MB/sec warmup 25 sec latency 19999.345 ms 2024-08-19T01:46:54.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 13.80 MB/sec warmup 26 sec latency 20999.443 ms 2024-08-19T01:46:55.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:55 smithi123 ceph-mon[21829]: pgmap v427: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 58 op/s 2024-08-19T01:46:55.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:55 smithi161 ceph-mon[21757]: pgmap v427: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 58 op/s 2024-08-19T01:46:55.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:55 smithi064 ceph-mon[18515]: pgmap v427: 129 pgs: 129 active+clean; 623 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 58 op/s 2024-08-19T01:46:55.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 13.28 MB/sec warmup 27 sec latency 21999.506 ms 2024-08-19T01:46:55.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:46:56.173 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:46:56.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.662+0000 7f0c8060e640 1 -- 172.21.15.64:0/2523181165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781067b0 msgr2=0x7f0c78106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:56.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.662+0000 7f0c8060e640 1 --2- 172.21.15.64:0/2523181165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781067b0 0x7f0c78106b90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009930 tx=0x7f0c7402a420 comp rx=0 tx=0).stop 2024-08-19T01:46:56.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.662+0000 7f0c8060e640 1 -- 172.21.15.64:0/2523181165 shutdown_connections 2024-08-19T01:46:56.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.662+0000 7f0c8060e640 1 --2- 172.21.15.64:0/2523181165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c78101150 0x7f0c7810d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.662+0000 7f0c8060e640 1 --2- 172.21.15.64:0/2523181165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c781007b0 0x7f0c78100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.662+0000 7f0c8060e640 1 --2- 172.21.15.64:0/2523181165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781067b0 0x7f0c78106b90 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.662+0000 7f0c8060e640 1 -- 172.21.15.64:0/2523181165 >> 172.21.15.64:0/2523181165 conn(0x7f0c780fc4d0 msgr2=0x7f0c780fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.663+0000 7f0c8060e640 1 -- 172.21.15.64:0/2523181165 shutdown_connections 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.663+0000 7f0c8060e640 1 -- 172.21.15.64:0/2523181165 wait complete. 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.663+0000 7f0c8060e640 1 Processor -- start 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.663+0000 7f0c8060e640 1 -- start start 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.663+0000 7f0c8060e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781007b0 0x7f0c78198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.663+0000 7f0c8060e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c78101150 0x7f0c78198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c8060e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c781067b0 0x7f0c7819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c8060e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78110bb0 con 0x7f0c781007b0 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c8060e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78110a30 con 0x7f0c781067b0 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c8060e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78110d30 con 0x7f0c78101150 2024-08-19T01:46:56.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781007b0 0x7f0c78198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781007b0 0x7f0c78198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50058/0 (socket says 172.21.15.64:50058) 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 -- 172.21.15.64:0/1722678022 learned_addr learned my addr 172.21.15.64:0/1722678022 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7dba9640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c78101150 0x7f0c78198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7ebab640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c781067b0 0x7f0c7819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 -- 172.21.15.64:0/1722678022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c78101150 msgr2=0x7f0c78198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c78101150 0x7f0c78198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 -- 172.21.15.64:0/1722678022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c781067b0 msgr2=0x7f0c7819cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c781067b0 0x7f0c7819cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 -- 172.21.15.64:0/1722678022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7819d1e0 con 0x7f0c781007b0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c7e3aa640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781007b0 0x7f0c78198230 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009230 tx=0x7f0c74004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c6f7fe640 1 -- 172.21.15.64:0/1722678022 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7403f030 con 0x7f0c781007b0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7819d470 con 0x7f0c781007b0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c6f7fe640 1 -- 172.21.15.64:0/1722678022 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c74003ad0 con 0x7f0c781007b0 2024-08-19T01:46:56.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c6f7fe640 1 -- 172.21.15.64:0/1722678022 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74035070 con 0x7f0c781007b0 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.664+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c781a4db0 con 0x7f0c781007b0 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.665+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c78101ef0 con 0x7f0c781007b0 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.666+0000 7f0c6f7fe640 1 -- 172.21.15.64:0/1722678022 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0c74042030 con 0x7f0c781007b0 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.667+0000 7f0c6f7fe640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0c640a2cf0 0x7f0c640a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.667+0000 7f0c7dba9640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0c640a2cf0 0x7f0c640a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.667+0000 7f0c6f7fe640 1 -- 172.21.15.64:0/1722678022 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f0c740efd30 con 0x7f0c781007b0 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.668+0000 7f0c7dba9640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0c640a2cf0 0x7f0c640a51b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0c78104f40 tx=0x7f0c6800b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:56.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.671+0000 7f0c6f7fe640 1 -- 172.21.15.64:0/1722678022 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f0c740bc540 con 0x7f0c781007b0 2024-08-19T01:46:56.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 12.81 MB/sec warmup 28 sec latency 22999.556 ms 2024-08-19T01:46:56.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.882+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c78109d30 con 0x7f0c781007b0 2024-08-19T01:46:56.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.883+0000 7f0c6f7fe640 1 -- 172.21.15.64:0/1722678022 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11924 (secure 0 0 0) 0x7f0c740bbe40 con 0x7f0c781007b0 2024-08-19T01:46:56.884 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:46:56.884 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"btime":"2024-08-19T01:46:31:794441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:46:31.560051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:46:56.884 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-08-19T01:46:56.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0c640a2cf0 msgr2=0x7f0c640a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:56.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0c640a2cf0 0x7f0c640a51b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0c78104f40 tx=0x7f0c6800b390 comp rx=0 tx=0).stop 2024-08-19T01:46:56.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781007b0 msgr2=0x7f0c78198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:56.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781007b0 0x7f0c78198230 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009230 tx=0x7f0c74004270 comp rx=0 tx=0).stop 2024-08-19T01:46:56.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 shutdown_connections 2024-08-19T01:46:56.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f0c640a2cf0 0x7f0c640a51b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c781067b0 0x7f0c7819cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c78101150 0x7f0c78198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 --2- 172.21.15.64:0/1722678022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c781007b0 0x7f0c78198230 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:56.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 >> 172.21.15.64:0/1722678022 conn(0x7f0c780fc4d0 msgr2=0x7f0c7806a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:56.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 shutdown_connections 2024-08-19T01:46:56.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:56.885+0000 7f0c8060e640 1 -- 172.21.15.64:0/1722678022 wait complete. 2024-08-19T01:46:57.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:57 smithi064 ceph-mon[18515]: pgmap v428: 129 pgs: 129 active+clean; 783 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 17 MiB/s wr, 165 op/s 2024-08-19T01:46:57.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:57 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1722678022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:46:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:57 smithi123 ceph-mon[21829]: pgmap v428: 129 pgs: 129 active+clean; 783 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 17 MiB/s wr, 165 op/s 2024-08-19T01:46:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:57 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1722678022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:46:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:57 smithi161 ceph-mon[21757]: pgmap v428: 129 pgs: 129 active+clean; 783 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 17 MiB/s wr, 165 op/s 2024-08-19T01:46:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:57 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1722678022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:46:57.647 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 9bec36f3 --release --await' 2024-08-19T01:46:57.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 9bec36f3 --release --await 2024-08-19T01:46:57.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3406 12.37 MB/sec warmup 29 sec latency 23999.612 ms 2024-08-19T01:46:57.903 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 -- 172.21.15.64:0/22199190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01018b0 msgr2=0x7f04f0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 --2- 172.21.15.64:0/22199190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01018b0 0x7f04f0101d10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f04e0009930 tx=0x7f04e002a420 comp rx=0 tx=0).stop 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 -- 172.21.15.64:0/22199190 shutdown_connections 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 --2- 172.21.15.64:0/22199190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04f0102250 0x7f04f010e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 --2- 172.21.15.64:0/22199190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01018b0 0x7f04f0101d10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 --2- 172.21.15.64:0/22199190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04f01078b0 0x7f04f0107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 -- 172.21.15.64:0/22199190 >> 172.21.15.64:0/22199190 conn(0x7f04f00fd5f0 msgr2=0x7f04f00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 -- 172.21.15.64:0/22199190 shutdown_connections 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.385+0000 7f04f7278640 1 -- 172.21.15.64:0/22199190 wait complete. 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 Processor -- start 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 -- start start 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04f01018b0 0x7f04f0074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04f0102250 0x7f04f0074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01078b0 0x7f04f0074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04f0111cd0 con 0x7f04f01078b0 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04f0111b50 con 0x7f04f0102250 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f7278640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04f0111e50 con 0x7f04f01018b0 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f5815640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01078b0 0x7f04f0074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f5815640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01078b0 0x7f04f0074ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50080/0 (socket says 172.21.15.64:50080) 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f5815640 1 -- 172.21.15.64:0/669952322 learned_addr learned my addr 172.21.15.64:0/669952322 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.386+0000 7f04f4813640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04f0102250 0x7f04f0074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f5014640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04f01018b0 0x7f04f0074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f5815640 1 -- 172.21.15.64:0/669952322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04f01018b0 msgr2=0x7f04f0074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f5815640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04f01018b0 0x7f04f0074220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f5815640 1 -- 172.21.15.64:0/669952322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04f0102250 msgr2=0x7f04f0074760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f5815640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04f0102250 0x7f04f0074760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f5815640 1 -- 172.21.15.64:0/669952322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f01a66a0 con 0x7f04f01078b0 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f5815640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01078b0 0x7f04f0074ca0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f04e800b970 tx=0x7f04e800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04e67fc640 1 -- 172.21.15.64:0/669952322 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e8019070 con 0x7f04f01078b0 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04e67fc640 1 -- 172.21.15.64:0/669952322 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e800cb40 con 0x7f04f01078b0 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04f01a6990 con 0x7f04f01078b0 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04e67fc640 1 -- 172.21.15.64:0/669952322 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e8007650 con 0x7f04f01078b0 2024-08-19T01:46:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.387+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04f01a6ea0 con 0x7f04f01078b0 2024-08-19T01:46:58.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.390+0000 7f04e67fc640 1 -- 172.21.15.64:0/669952322 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f04e801c070 con 0x7f04f01078b0 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.390+0000 7f04f9676740 1 Processor -- start 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.390+0000 7f04f9676740 1 -- start start 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f9676740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616a990 0x55fbe616ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f9676740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616b2b0 0x55fbe6175630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f9676740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe6177f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f9676740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fbe60d0c60 con 0x55fbe6175b70 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f9676740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fbe6068b90 con 0x55fbe616a990 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f9676740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fbe60ccc20 con 0x55fbe616b2b0 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe6177f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe6177f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50084/0 (socket says 172.21.15.64:50084) 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 -- 172.21.15.64:0/3575783376 learned_addr learned my addr 172.21.15.64:0/3575783376 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:58.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616a990 0x55fbe616ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f5014640 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616b2b0 0x55fbe6175630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04e67fc640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04dc0a2d70 0x7f04dc0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 -- 172.21.15.64:0/3575783376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616b2b0 msgr2=0x55fbe6175630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616b2b0 0x55fbe6175630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 -- 172.21.15.64:0/3575783376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616a990 msgr2=0x55fbe616ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616a990 0x55fbe616ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 -- 172.21.15.64:0/3575783376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe5de0580 con 0x55fbe6175b70 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f5815640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04dc0a2d70 0x7f04dc0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f5014640 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616b2b0 0x55fbe6175630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.391+0000 7f04f4813640 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe6177f60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f04e00096c0 tx=0x7f04e002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eae95db35270513 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.392+0000 7f04e67fc640 1 -- 172.21.15.64:0/669952322 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f04e80cecd0 con 0x7f04f01078b0 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.392+0000 7f04db7fe640 1 -- 172.21.15.64:0/3575783376 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0035070 con 0x55fbe6175b70 2024-08-19T01:46:58.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.392+0000 7f04db7fe640 1 -- 172.21.15.64:0/3575783376 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e0005020 con 0x55fbe6175b70 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.392+0000 7f04db7fe640 1 -- 172.21.15.64:0/3575783376 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0004a00 con 0x55fbe6175b70 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.392+0000 7f04f9676740 1 -- 172.21.15.64:0/3575783376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 msgr2=0x55fbe6177f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.392+0000 7f04f5815640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04dc0a2d70 0x7f04dc0a5230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f04e80cc1d0 tx=0x7f04e8004a40 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.392+0000 7f04f9676740 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe6177f60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f04e00096c0 tx=0x7f04e002a420 comp rx=0 tx=0).stop 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- 172.21.15.64:0/3575783376 shutdown_connections 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe6177f60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616b2b0 0x55fbe6175630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 --2- 172.21.15.64:0/3575783376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616a990 0x55fbe616ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- 172.21.15.64:0/3575783376 >> 172.21.15.64:0/3575783376 conn(0x55fbe6166f20 msgr2=0x55fbe6167300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:58.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- 172.21.15.64:0/3575783376 shutdown_connections 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- 172.21.15.64:0/3575783376 wait complete. 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- start start 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 Processor -- start 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616a990 0x55fbe6172a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe6172f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe61734d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fbe6178740 con 0x55fbe6175b70 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fbe606e7c0 con 0x55fbe616b2b0 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.393+0000 7f04f9676740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fbe61788c0 con 0x55fbe616a990 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f9676740 10 client.0 _ll_register_callbacks cb 0x55fbe6077f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe61734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe61734d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50096/0 (socket says 172.21.15.64:50096) 2024-08-19T01:46:58.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 -- 172.21.15.64:0/2303323890 learned_addr learned my addr 172.21.15.64:0/2303323890 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f5014640 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe6172f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616a990 0x55fbe6172a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 -- 172.21.15.64:0/2303323890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616a990 msgr2=0x55fbe6172a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616a990 0x55fbe6172a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 -- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 msgr2=0x55fbe6172f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe6172f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 -- 172.21.15.64:0/2303323890 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe5eaa960 con 0x55fbe6175b70 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f4813640 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe61734d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f04e002a940 tx=0x7f04e0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1d9fb22749e943 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04d8ff9640 1 -- 172.21.15.64:0/2303323890 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0035070 con 0x55fbe6175b70 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04d8ff9640 1 -- 172.21.15.64:0/2303323890 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f04e0004770 con 0x55fbe6175b70 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04e0005020 con 0x55fbe6175b70 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04d8ff9640 1 -- 172.21.15.64:0/2303323890 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f04e005c050 con 0x55fbe6175b70 2024-08-19T01:46:58.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f9676740 10 client.16041 fetch_fsmap learned FSMap version 30 2024-08-19T01:46:58.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.394+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fbe61847b0 con 0x55fbe6175b70 2024-08-19T01:46:58.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04d8ff9640 1 -- 172.21.15.64:0/2303323890 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e0005020 con 0x55fbe6175b70 2024-08-19T01:46:58.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04d8ff9640 1 -- 172.21.15.64:0/2303323890 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f04e003db80 con 0x55fbe6175b70 2024-08-19T01:46:58.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04f9676740 10 client.16041 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04f9676740 10 client.16041 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04f9676740 20 client.16041 populate_metadata read hostname 'smithi064' 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04f9676740 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe6185b00 0x55fbe6187ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04f9676740 4 client.16041 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbe60de1d0 con 0x55fbe6185b00 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.395+0000 7f04f5014640 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe6185b00 0x55fbe6187ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.396+0000 7f04f5014640 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe6185b00 0x55fbe6187ef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ce58bd990f4b68 server_cookie=a03215356f56f1d4 in_seq=0 out_seq=0 2024-08-19T01:46:58.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.396+0000 7f04d8ff9640 10 client.16041 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:58.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.396+0000 7f04d8ff9640 1 -- 172.21.15.64:0/2303323890 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fbe60de1d0 con 0x55fbe6185b00 2024-08-19T01:46:58.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.396+0000 7f04d8ff9640 10 client.16041 handle_command_reply: tid=0 2024-08-19T01:46:58.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.396+0000 7f04f9676740 1 client.16041 shutdown 2024-08-19T01:46:58.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.396+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 msgr2=0x55fbe61734d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.396+0000 7f04f9676740 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe61734d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f04e002a940 tx=0x7f04e0005c90 comp rx=0 tx=0).stop 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 shutdown_connections 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe6175b70 0x55fbe61734d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe6185b00 0x55fbe6187ef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe6172f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 --2- 172.21.15.64:0/2303323890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe616a990 0x55fbe6172a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 >> 172.21.15.64:0/2303323890 conn(0x55fbe6166f20 msgr2=0x55fbe616cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04d8ff9640 0 client.16041 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 shutdown_connections 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 1 -- 172.21.15.64:0/2303323890 wait complete. 2024-08-19T01:46:58.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.397+0000 7f04f9676740 20 client.16041 trim_cache size 0 max 16384 2024-08-19T01:46:58.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 Processor -- start 2024-08-19T01:46:58.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 -- start start 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe616caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe618ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 0x55fbe618b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04f01b3940 con 0x55fbe616a990 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04f01b1210 con 0x55fbe616b2b0 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f9676740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04f01af080 con 0x55fbe6175b70 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe616caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe616caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50112/0 (socket says 172.21.15.64:50112) 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.414+0000 7f04f4813640 1 -- 172.21.15.64:0/1479374523 learned_addr learned my addr 172.21.15.64:0/1479374523 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04f4813640 1 -- 172.21.15.64:0/1479374523 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 msgr2=0x55fbe618b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04f4813640 1 --2- 172.21.15.64:0/1479374523 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 0x55fbe618b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04f4813640 1 -- 172.21.15.64:0/1479374523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 msgr2=0x55fbe618ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04f4813640 1 --2- 172.21.15.64:0/1479374523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe618ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04f4813640 1 -- 172.21.15.64:0/1479374523 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe6170fa0 con 0x55fbe616a990 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04f4813640 1 --2- 172.21.15.64:0/1479374523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe616caf0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f04e0032c00 tx=0x7f04e0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da330ce3325d9492 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04daffd640 1 -- 172.21.15.64:0/1479374523 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0065070 con 0x55fbe616a990 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04daffd640 1 -- 172.21.15.64:0/1479374523 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e0053070 con 0x55fbe616a990 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.415+0000 7f04daffd640 1 -- 172.21.15.64:0/1479374523 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0035040 con 0x55fbe616a990 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 -- 172.21.15.64:0/1479374523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 msgr2=0x55fbe616caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 --2- 172.21.15.64:0/1479374523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe616caf0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f04e0032c00 tx=0x7f04e0005a20 comp rx=0 tx=0).stop 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 -- 172.21.15.64:0/1479374523 shutdown_connections 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 --2- 172.21.15.64:0/1479374523 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 0x55fbe618b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 --2- 172.21.15.64:0/1479374523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe618ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 --2- 172.21.15.64:0/1479374523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe616caf0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 -- 172.21.15.64:0/1479374523 >> 172.21.15.64:0/1479374523 conn(0x55fbe6166f20 msgr2=0x55fbe616c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 -- 172.21.15.64:0/1479374523 shutdown_connections 2024-08-19T01:46:58.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 -- 172.21.15.64:0/1479374523 wait complete. 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 -- start start 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.416+0000 7f04f9676740 1 Processor -- start 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe6188f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe61894d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 0x55fbe61ef440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04f01af6d0 con 0x55fbe616a990 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04f01b0b40 con 0x55fbe616b2b0 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04f01afda0 con 0x55fbe6175b70 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 10 client.0 _ll_register_callbacks cb 0x7f04e00319d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe6188f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe6188f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50120/0 (socket says 172.21.15.64:50120) 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 -- 172.21.15.64:0/1116829126 learned_addr learned my addr 172.21.15.64:0/1116829126 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:46:58.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f5014640 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe61894d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 0x55fbe61ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 -- 172.21.15.64:0/1116829126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 msgr2=0x55fbe61ef440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 0x55fbe61ef440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 -- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 msgr2=0x55fbe61894d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe61894d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 -- 172.21.15.64:0/1116829126 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe61ef980 con 0x55fbe616a990 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f4813640 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe6188f90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f04e0063040 tx=0x7f04e0031960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5363833c071f42a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04d8ff9640 1 -- 172.21.15.64:0/1116829126 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0065040 con 0x55fbe616a990 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04d8ff9640 1 -- 172.21.15.64:0/1116829126 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f04e003c490 con 0x55fbe616a990 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.417+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04e0053070 con 0x55fbe616a990 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04d8ff9640 1 -- 172.21.15.64:0/1116829126 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f04e0035040 con 0x55fbe616a990 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 10 client.16047 fetch_fsmap learned FSMap version 30 2024-08-19T01:46:58.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fbe61f0c00 con 0x55fbe616a990 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04d8ff9640 1 -- 172.21.15.64:0/1116829126 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e0053070 con 0x55fbe616a990 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04d8ff9640 1 -- 172.21.15.64:0/1116829126 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f04e003d930 con 0x55fbe616a990 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 10 client.16047 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 10 client.16047 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 20 client.16047 populate_metadata read hostname 'smithi064' 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe61f1000 0x55fbe61f3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 4 client.16047 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9bec36f3", "release": true, "await": true}] 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9bec36f3", "release": true, "await": true}) -- 0x55fbe60d86e0 con 0x55fbe61f1000 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.418+0000 7f04f5014640 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe61f1000 0x55fbe61f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.419+0000 7f04f5014640 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe61f1000 0x55fbe61f3420 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83f5d2ef700198fc server_cookie=3580b92855f5f963 in_seq=0 out_seq=0 2024-08-19T01:46:58.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.419+0000 7f04d8ff9640 10 client.16047 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04d8ff9640 1 -- 172.21.15.64:0/1116829126 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fbe60d86e0 con 0x55fbe61f1000 2024-08-19T01:46:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04d8ff9640 10 client.16047 handle_command_reply: tid=0 2024-08-19T01:46:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 client.16047 shutdown 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 msgr2=0x55fbe6188f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe6188f90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f04e0063040 tx=0x7f04e0031960 comp rx=0 tx=0).stop 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 shutdown_connections 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fbe6175b70 0x55fbe61ef440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55fbe61f1000 0x55fbe61f3420 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fbe616b2b0 0x55fbe61894d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 --2- 172.21.15.64:0/1116829126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbe616a990 0x55fbe6188f90 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 >> 172.21.15.64:0/1116829126 conn(0x55fbe6166f20 msgr2=0x55fbe6181360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04d8ff9640 0 client.16047 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 shutdown_connections 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 1 -- 172.21.15.64:0/1116829126 wait complete. 2024-08-19T01:46:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.479+0000 7f04f9676740 20 client.16047 trim_cache size 0 max 16384 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 48, 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "9bec36f3": { 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 48, 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:46:58.483 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 145.1, 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:58.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:46:58.485 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:46:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04dc0a2d70 msgr2=0x7f04dc0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04dc0a2d70 0x7f04dc0a5230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f04e80cc1d0 tx=0x7f04e8004a40 comp rx=0 tx=0).stop 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01078b0 msgr2=0x7f04f0074ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01078b0 0x7f04f0074ca0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f04e800b970 tx=0x7f04e800be30 comp rx=0 tx=0).stop 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 shutdown_connections 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04dc0a2d70 0x7f04dc0a5230 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04f01078b0 0x7f04f0074ca0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04f0102250 0x7f04f0074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 --2- 172.21.15.64:0/669952322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04f01018b0 0x7f04f0074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 >> 172.21.15.64:0/669952322 conn(0x7f04f00fd5f0 msgr2=0x7f04f00fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 shutdown_connections 2024-08-19T01:46:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:46:58.480+0000 7f04f7278640 1 -- 172.21.15.64:0/669952322 wait complete. 2024-08-19T01:46:58.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 3583 12.50 MB/sec warmup 30 sec latency 24779.505 ms 2024-08-19T01:46:59.212 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9bec36f3', total seconds elapsed: 28.4 2024-08-19T01:46:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:46:59 smithi123 ceph-mon[21829]: pgmap v429: 129 pgs: 129 active+clean; 783 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 16 MiB/s wr, 135 op/s 2024-08-19T01:46:59.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:46:59 smithi161 ceph-mon[21757]: pgmap v429: 129 pgs: 129 active+clean; 783 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 16 MiB/s wr, 135 op/s 2024-08-19T01:46:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:46:59 smithi064 ceph-mon[18515]: pgmap v429: 129 pgs: 129 active+clean; 783 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 16 MiB/s wr, 135 op/s 2024-08-19T01:46:59.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4175 13.17 MB/sec warmup 31 sec latency 43.956 ms 2024-08-19T01:47:00.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4658 13.45 MB/sec warmup 32 sec latency 88.819 ms 2024-08-19T01:47:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:01 smithi123 ceph-mon[21829]: pgmap v430: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 194 op/s 2024-08-19T01:47:01.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:01 smithi161 ceph-mon[21757]: pgmap v430: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 194 op/s 2024-08-19T01:47:01.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:01 smithi064 ceph-mon[18515]: pgmap v430: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 194 op/s 2024-08-19T01:47:01.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4734 13.10 MB/sec warmup 33 sec latency 823.765 ms 2024-08-19T01:47:02.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4734 12.72 MB/sec warmup 34 sec latency 1823.887 ms 2024-08-19T01:47:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:03 smithi123 ceph-mon[21829]: pgmap v431: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 16 MiB/s wr, 165 op/s 2024-08-19T01:47:03.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:03 smithi161 ceph-mon[21757]: pgmap v431: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 16 MiB/s wr, 165 op/s 2024-08-19T01:47:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:03 smithi064 ceph-mon[18515]: pgmap v431: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 16 MiB/s wr, 165 op/s 2024-08-19T01:47:03.718 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4734 12.35 MB/sec warmup 35 sec latency 2823.973 ms 2024-08-19T01:47:04.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4734 12.01 MB/sec warmup 36 sec latency 3824.088 ms 2024-08-19T01:47:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:05 smithi123 ceph-mon[21829]: pgmap v432: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 16 MiB/s wr, 165 op/s 2024-08-19T01:47:05.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:05 smithi161 ceph-mon[21757]: pgmap v432: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 16 MiB/s wr, 165 op/s 2024-08-19T01:47:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:05 smithi064 ceph-mon[18515]: pgmap v432: 129 pgs: 129 active+clean; 791 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 16 MiB/s wr, 165 op/s 2024-08-19T01:47:05.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4779 11.75 MB/sec warmup 37 sec latency 4066.375 ms 2024-08-19T01:47:06.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 4822 11.47 MB/sec warmup 38 sec latency 1555.824 ms 2024-08-19T01:47:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:07 smithi123 ceph-mon[21829]: pgmap v433: 129 pgs: 129 active+clean; 824 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 19 MiB/s wr, 210 op/s 2024-08-19T01:47:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:07 smithi161 ceph-mon[21757]: pgmap v433: 129 pgs: 129 active+clean; 824 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 19 MiB/s wr, 210 op/s 2024-08-19T01:47:07.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:07 smithi064 ceph-mon[18515]: pgmap v433: 129 pgs: 129 active+clean; 824 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 19 MiB/s wr, 210 op/s 2024-08-19T01:47:07.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 5261 11.76 MB/sec warmup 39 sec latency 108.904 ms 2024-08-19T01:47:08.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 5646 12.02 MB/sec warmup 40 sec latency 251.531 ms 2024-08-19T01:47:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:09 smithi123 ceph-mon[21829]: pgmap v434: 129 pgs: 129 active+clean; 824 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 103 op/s 2024-08-19T01:47:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:09 smithi161 ceph-mon[21757]: pgmap v434: 129 pgs: 129 active+clean; 824 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 103 op/s 2024-08-19T01:47:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:09 smithi064 ceph-mon[18515]: pgmap v434: 129 pgs: 129 active+clean; 824 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 103 op/s 2024-08-19T01:47:09.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 6127 12.12 MB/sec warmup 41 sec latency 194.383 ms 2024-08-19T01:47:10.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 7029 12.79 MB/sec warmup 42 sec latency 47.965 ms 2024-08-19T01:47:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:11 smithi123 ceph-mon[21829]: pgmap v435: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.5 MiB/s wr, 343 op/s 2024-08-19T01:47:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:11 smithi161 ceph-mon[21757]: pgmap v435: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.5 MiB/s wr, 343 op/s 2024-08-19T01:47:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:11 smithi064 ceph-mon[18515]: pgmap v435: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.5 MiB/s wr, 343 op/s 2024-08-19T01:47:11.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 7886 14.02 MB/sec warmup 43 sec latency 49.558 ms 2024-08-19T01:47:12.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 8396 14.52 MB/sec warmup 44 sec latency 83.889 ms 2024-08-19T01:47:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:13 smithi123 ceph-mon[21829]: pgmap v436: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.8 MiB/s wr, 285 op/s 2024-08-19T01:47:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:13 smithi123 ceph-mon[21829]: osdmap e109: 12 total, 12 up, 12 in 2024-08-19T01:47:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:13 smithi161 ceph-mon[21757]: pgmap v436: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.8 MiB/s wr, 285 op/s 2024-08-19T01:47:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:13 smithi161 ceph-mon[21757]: osdmap e109: 12 total, 12 up, 12 in 2024-08-19T01:47:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:13 smithi064 ceph-mon[18515]: pgmap v436: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.8 MiB/s wr, 285 op/s 2024-08-19T01:47:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:13 smithi064 ceph-mon[18515]: osdmap e109: 12 total, 12 up, 12 in 2024-08-19T01:47:13.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 8843 14.61 MB/sec warmup 45 sec latency 58.267 ms 2024-08-19T01:47:14.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 9336 14.76 MB/sec warmup 46 sec latency 107.692 ms 2024-08-19T01:47:15.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:15 smithi123 ceph-mon[21829]: pgmap v438: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.2 MiB/s wr, 342 op/s 2024-08-19T01:47:15.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:15 smithi161 ceph-mon[21757]: pgmap v438: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.2 MiB/s wr, 342 op/s 2024-08-19T01:47:15.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:15 smithi064 ceph-mon[18515]: pgmap v438: 129 pgs: 129 active+clean; 852 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 8.2 MiB/s wr, 342 op/s 2024-08-19T01:47:15.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 9882 15.05 MB/sec warmup 47 sec latency 59.498 ms 2024-08-19T01:47:16.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 10572 15.28 MB/sec warmup 48 sec latency 58.825 ms 2024-08-19T01:47:17.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:17 smithi123 ceph-mon[21829]: pgmap v439: 129 pgs: 129 active+clean; 874 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.1 MiB/s wr, 491 op/s 2024-08-19T01:47:17.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:17 smithi161 ceph-mon[21757]: pgmap v439: 129 pgs: 129 active+clean; 874 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.1 MiB/s wr, 491 op/s 2024-08-19T01:47:17.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:17 smithi064 ceph-mon[18515]: pgmap v439: 129 pgs: 129 active+clean; 874 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.1 MiB/s wr, 491 op/s 2024-08-19T01:47:17.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 11452 16.03 MB/sec warmup 49 sec latency 54.899 ms 2024-08-19T01:47:18.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 12220 16.81 MB/sec warmup 50 sec latency 56.613 ms 2024-08-19T01:47:19.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 12728 17.08 MB/sec warmup 51 sec latency 85.781 ms 2024-08-19T01:47:20.117 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:19 smithi123 ceph-mon[21829]: pgmap v440: 129 pgs: 129 active+clean; 874 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.1 MiB/s wr, 491 op/s 2024-08-19T01:47:20.117 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:19 smithi123 ceph-mon[21829]: osdmap e110: 12 total, 12 up, 12 in 2024-08-19T01:47:20.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:19 smithi064 ceph-mon[18515]: pgmap v440: 129 pgs: 129 active+clean; 874 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.1 MiB/s wr, 491 op/s 2024-08-19T01:47:20.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:19 smithi064 ceph-mon[18515]: osdmap e110: 12 total, 12 up, 12 in 2024-08-19T01:47:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:19 smithi161 ceph-mon[21757]: pgmap v440: 129 pgs: 129 active+clean; 874 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.1 MiB/s wr, 491 op/s 2024-08-19T01:47:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:19 smithi161 ceph-mon[21757]: osdmap e110: 12 total, 12 up, 12 in 2024-08-19T01:47:20.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 13145 17.09 MB/sec warmup 52 sec latency 80.859 ms 2024-08-19T01:47:21.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 13618 17.19 MB/sec warmup 53 sec latency 162.743 ms 2024-08-19T01:47:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:21 smithi123 ceph-mon[21829]: pgmap v442: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 498 op/s 2024-08-19T01:47:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:21 smithi161 ceph-mon[21757]: pgmap v442: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 498 op/s 2024-08-19T01:47:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:21 smithi064 ceph-mon[18515]: pgmap v442: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 498 op/s 2024-08-19T01:47:22.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 14073 17.20 MB/sec warmup 54 sec latency 284.894 ms 2024-08-19T01:47:23.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 14718 17.55 MB/sec warmup 55 sec latency 144.271 ms 2024-08-19T01:47:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:23 smithi123 ceph-mon[21829]: pgmap v443: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.3 MiB/s wr, 402 op/s 2024-08-19T01:47:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:23 smithi161 ceph-mon[21757]: pgmap v443: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.3 MiB/s wr, 402 op/s 2024-08-19T01:47:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:23 smithi064 ceph-mon[18515]: pgmap v443: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.3 MiB/s wr, 402 op/s 2024-08-19T01:47:24.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 15331 17.81 MB/sec warmup 56 sec latency 134.476 ms 2024-08-19T01:47:25.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 15987 18.10 MB/sec warmup 57 sec latency 172.220 ms 2024-08-19T01:47:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:25 smithi123 ceph-mon[21829]: pgmap v444: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.2 MiB/s wr, 398 op/s 2024-08-19T01:47:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:25 smithi161 ceph-mon[21757]: pgmap v444: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.2 MiB/s wr, 398 op/s 2024-08-19T01:47:26.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:25 smithi064 ceph-mon[18515]: pgmap v444: 129 pgs: 129 active+clean; 897 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.2 MiB/s wr, 398 op/s 2024-08-19T01:47:26.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 16653 18.49 MB/sec warmup 58 sec latency 66.970 ms 2024-08-19T01:47:27.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 17250 18.74 MB/sec warmup 59 sec latency 67.778 ms 2024-08-19T01:47:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:27 smithi123 ceph-mon[21829]: pgmap v445: 129 pgs: 129 active+clean; 851 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.6 MiB/s wr, 407 op/s 2024-08-19T01:47:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:27 smithi161 ceph-mon[21757]: pgmap v445: 129 pgs: 129 active+clean; 851 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.6 MiB/s wr, 407 op/s 2024-08-19T01:47:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:27 smithi064 ceph-mon[18515]: pgmap v445: 129 pgs: 129 active+clean; 851 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.6 MiB/s wr, 407 op/s 2024-08-19T01:47:28.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 17761 18.98 MB/sec warmup 60 sec latency 42.716 ms 2024-08-19T01:47:29.719 INFO:tasks.workunit.client.0.smithi064.stdout: 10 18093 18.89 MB/sec warmup 61 sec latency 88.573 ms 2024-08-19T01:47:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:29 smithi123 ceph-mon[21829]: pgmap v446: 129 pgs: 129 active+clean; 851 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.6 MiB/s wr, 407 op/s 2024-08-19T01:47:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:29 smithi161 ceph-mon[21757]: pgmap v446: 129 pgs: 129 active+clean; 851 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.6 MiB/s wr, 407 op/s 2024-08-19T01:47:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:29 smithi064 ceph-mon[18515]: pgmap v446: 129 pgs: 129 active+clean; 851 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.6 MiB/s wr, 407 op/s 2024-08-19T01:47:30.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 18715 19.01 MB/sec warmup 62 sec latency 81.584 ms 2024-08-19T01:47:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:30 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:47:31.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:30 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:47:31.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:30 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:47:31.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 19485 19.35 MB/sec warmup 63 sec latency 114.962 ms 2024-08-19T01:47:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:31 smithi123 ceph-mon[21829]: pgmap v447: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.2 MiB/s wr, 536 op/s 2024-08-19T01:47:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:31 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:47:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:31 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:47:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:31 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:47:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:31 smithi161 ceph-mon[21757]: pgmap v447: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.2 MiB/s wr, 536 op/s 2024-08-19T01:47:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:31 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:47:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:31 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:47:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:31 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:47:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:31 smithi064 ceph-mon[18515]: pgmap v447: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.2 MiB/s wr, 536 op/s 2024-08-19T01:47:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:31 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:47:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:31 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:47:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:31 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:47:32.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 20139 19.66 MB/sec warmup 64 sec latency 54.667 ms 2024-08-19T01:47:33.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 20731 19.90 MB/sec warmup 65 sec latency 73.186 ms 2024-08-19T01:47:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:33 smithi123 ceph-mon[21829]: pgmap v448: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.5 MiB/s wr, 340 op/s 2024-08-19T01:47:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:33 smithi161 ceph-mon[21757]: pgmap v448: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.5 MiB/s wr, 340 op/s 2024-08-19T01:47:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:33 smithi064 ceph-mon[18515]: pgmap v448: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.5 MiB/s wr, 340 op/s 2024-08-19T01:47:34.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 21321 20.10 MB/sec warmup 66 sec latency 61.844 ms 2024-08-19T01:47:35.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 21882 20.31 MB/sec warmup 67 sec latency 52.323 ms 2024-08-19T01:47:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:35 smithi123 ceph-mon[21829]: pgmap v449: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.5 MiB/s wr, 340 op/s 2024-08-19T01:47:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:35 smithi161 ceph-mon[21757]: pgmap v449: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.5 MiB/s wr, 340 op/s 2024-08-19T01:47:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:35 smithi064 ceph-mon[18515]: pgmap v449: 129 pgs: 129 active+clean; 872 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.5 MiB/s wr, 340 op/s 2024-08-19T01:47:36.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 22260 20.26 MB/sec warmup 68 sec latency 93.052 ms 2024-08-19T01:47:37.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 22298 20.01 MB/sec warmup 69 sec latency 991.365 ms 2024-08-19T01:47:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:37 smithi123 ceph-mon[21829]: pgmap v450: 129 pgs: 129 active+clean; 894 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 523 op/s 2024-08-19T01:47:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:37 smithi161 ceph-mon[21757]: pgmap v450: 129 pgs: 129 active+clean; 894 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 523 op/s 2024-08-19T01:47:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:37 smithi064 ceph-mon[18515]: pgmap v450: 129 pgs: 129 active+clean; 894 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 523 op/s 2024-08-19T01:47:38.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 22356 19.74 MB/sec warmup 70 sec latency 1265.858 ms 2024-08-19T01:47:39.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 22628 19.57 MB/sec warmup 71 sec latency 129.904 ms 2024-08-19T01:47:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:39 smithi123 ceph-mon[21829]: pgmap v451: 129 pgs: 129 active+clean; 894 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.3 MiB/s wr, 346 op/s 2024-08-19T01:47:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:39 smithi161 ceph-mon[21757]: pgmap v451: 129 pgs: 129 active+clean; 894 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.3 MiB/s wr, 346 op/s 2024-08-19T01:47:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:39 smithi064 ceph-mon[18515]: pgmap v451: 129 pgs: 129 active+clean; 894 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.3 MiB/s wr, 346 op/s 2024-08-19T01:47:40.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 23137 19.66 MB/sec warmup 72 sec latency 91.824 ms 2024-08-19T01:47:41.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 24034 20.03 MB/sec warmup 73 sec latency 57.313 ms 2024-08-19T01:47:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:41 smithi123 ceph-mon[21829]: pgmap v452: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 761 op/s 2024-08-19T01:47:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:41 smithi161 ceph-mon[21757]: pgmap v452: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 761 op/s 2024-08-19T01:47:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:41 smithi064 ceph-mon[18515]: pgmap v452: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 16 MiB/s wr, 761 op/s 2024-08-19T01:47:42.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 24377 20.08 MB/sec warmup 74 sec latency 322.813 ms 2024-08-19T01:47:43.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 25017 20.36 MB/sec warmup 75 sec latency 63.027 ms 2024-08-19T01:47:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:43 smithi123 ceph-mon[21829]: pgmap v453: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 597 op/s 2024-08-19T01:47:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:43 smithi161 ceph-mon[21757]: pgmap v453: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 597 op/s 2024-08-19T01:47:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:43 smithi064 ceph-mon[18515]: pgmap v453: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 597 op/s 2024-08-19T01:47:44.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 25592 20.51 MB/sec warmup 76 sec latency 101.683 ms 2024-08-19T01:47:45.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 26172 20.56 MB/sec warmup 77 sec latency 75.558 ms 2024-08-19T01:47:45.827 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-19T01:47:45.828 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.65 2024-08-19T01:47:45.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:47:46.113 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:47:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:45 smithi123 ceph-mon[21829]: pgmap v454: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 598 op/s 2024-08-19T01:47:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:45 smithi161 ceph-mon[21757]: pgmap v454: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 598 op/s 2024-08-19T01:47:46.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:45 smithi064 ceph-mon[18515]: pgmap v454: 129 pgs: 129 active+clean; 922 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 598 op/s 2024-08-19T01:47:46.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 26604 20.61 MB/sec warmup 78 sec latency 209.775 ms 2024-08-19T01:47:46.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 -- 172.21.15.64:0/2171388675 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d90100b50 msgr2=0x7f2d90100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:46.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 --2- 172.21.15.64:0/2171388675 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d90100b50 0x7f2d90100f30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009900 tx=0x7f2d8002a480 comp rx=0 tx=0).stop 2024-08-19T01:47:46.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 -- 172.21.15.64:0/2171388675 shutdown_connections 2024-08-19T01:47:46.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 --2- 172.21.15.64:0/2171388675 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d90102c20 0x7f2d9010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:46.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 --2- 172.21.15.64:0/2171388675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d901022a0 0x7f2d901026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:46.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 --2- 172.21.15.64:0/2171388675 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d90100b50 0x7f2d90100f30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:46.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 -- 172.21.15.64:0/2171388675 >> 172.21.15.64:0/2171388675 conn(0x7f2d900fc500 msgr2=0x7f2d900fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:46.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 -- 172.21.15.64:0/2171388675 shutdown_connections 2024-08-19T01:47:46.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.756+0000 7f2d95026640 1 -- 172.21.15.64:0/2171388675 wait complete. 2024-08-19T01:47:46.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.757+0000 7f2d95026640 1 Processor -- start 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d95026640 1 -- start start 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d95026640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d90100b50 0x7f2d90198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d95026640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d901022a0 0x7f2d90198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d95026640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d90102c20 0x7f2d9019cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d95026640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2d90111dc0 con 0x7f2d901022a0 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d95026640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2d90111c40 con 0x7f2d90100b50 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d95026640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2d90111f40 con 0x7f2d90102c20 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d8e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d901022a0 0x7f2d90198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d8ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d90100b50 0x7f2d90198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d8e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d901022a0 0x7f2d90198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49930/0 (socket says 172.21.15.64:49930) 2024-08-19T01:47:46.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.758+0000 7f2d8e59c640 1 -- 172.21.15.64:0/3877382188 learned_addr learned my addr 172.21.15.64:0/3877382188 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8f59e640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d90102c20 0x7f2d9019cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8ed9d640 1 -- 172.21.15.64:0/3877382188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d90102c20 msgr2=0x7f2d9019cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8ed9d640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d90102c20 0x7f2d9019cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8ed9d640 1 -- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d901022a0 msgr2=0x7f2d90198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8ed9d640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d901022a0 0x7f2d90198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8ed9d640 1 -- 172.21.15.64:0/3877382188 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9019d1b0 con 0x7f2d90100b50 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8e59c640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d901022a0 0x7f2d90198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8f59e640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d90102c20 0x7f2d9019cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d8ed9d640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d90100b50 0x7f2d90198200 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2d80005940 tx=0x7f2d80003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d77fff640 1 -- 172.21.15.64:0/3877382188 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8003f070 con 0x7f2d90100b50 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9019d440 con 0x7f2d90100b50 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9010dfb0 con 0x7f2d90100b50 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.759+0000 7f2d77fff640 1 -- 172.21.15.64:0/3877382188 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d80031c30 con 0x7f2d90100b50 2024-08-19T01:47:46.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.760+0000 7f2d77fff640 1 -- 172.21.15.64:0/3877382188 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d80035030 con 0x7f2d90100b50 2024-08-19T01:47:46.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.761+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d7c005290 con 0x7f2d90100b50 2024-08-19T01:47:46.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.763+0000 7f2d77fff640 1 -- 172.21.15.64:0/3877382188 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2d80042070 con 0x7f2d90100b50 2024-08-19T01:47:46.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.763+0000 7f2d77fff640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2d780a2cf0 0x7f2d780a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:46.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.763+0000 7f2d8e59c640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2d780a2cf0 0x7f2d780a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:46.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.763+0000 7f2d77fff640 1 -- 172.21.15.64:0/3877382188 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f2d800f01c0 con 0x7f2d90100b50 2024-08-19T01:47:46.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.764+0000 7f2d8e59c640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2d780a2cf0 0x7f2d780a51b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2d900ffaf0 tx=0x7f2d84012040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:46.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:46.771+0000 7f2d77fff640 1 -- 172.21.15.64:0/3877382188 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f2d800bc9d0 con 0x7f2d90100b50 2024-08-19T01:47:47.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.018+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d7c005850 con 0x7f2d90100b50 2024-08-19T01:47:47.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.019+0000 7f2d77fff640 1 -- 172.21.15.64:0/3877382188 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11924 (secure 0 0 0) 0x7f2d800bc2d0 con 0x7f2d90100b50 2024-08-19T01:47:47.021 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:47:47.021 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"btime":"2024-08-19T01:46:31:794441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:46:31.560051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:47:47.021 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2d780a2cf0 msgr2=0x7f2d780a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2d780a2cf0 0x7f2d780a51b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2d900ffaf0 tx=0x7f2d84012040 comp rx=0 tx=0).stop 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d90100b50 msgr2=0x7f2d90198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d90100b50 0x7f2d90198200 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2d80005940 tx=0x7f2d80003c10 comp rx=0 tx=0).stop 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 shutdown_connections 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f2d780a2cf0 0x7f2d780a51b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d90102c20 0x7f2d9019cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d901022a0 0x7f2d90198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:47.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 --2- 172.21.15.64:0/3877382188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d90100b50 0x7f2d90198200 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:47.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.022+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 >> 172.21.15.64:0/3877382188 conn(0x7f2d900fc500 msgr2=0x7f2d900fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:47.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.023+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 shutdown_connections 2024-08-19T01:47:47.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:47.023+0000 7f2d95026640 1 -- 172.21.15.64:0/3877382188 wait complete. 2024-08-19T01:47:47.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27172 20.70 MB/sec warmup 79 sec latency 59.189 ms 2024-08-19T01:47:47.736 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 137.65 --await-for 45.0' 2024-08-19T01:47:47.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 137.65 --await-for 45.0 2024-08-19T01:47:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:47 smithi064 ceph-mon[18515]: pgmap v455: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 870 op/s 2024-08-19T01:47:47.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:47 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3877382188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:47:48.048 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:47:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:47 smithi123 ceph-mon[21829]: pgmap v455: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 870 op/s 2024-08-19T01:47:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:47 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3877382188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:47:48.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:47 smithi161 ceph-mon[21757]: pgmap v455: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 870 op/s 2024-08-19T01:47:48.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:47 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3877382188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:47:48.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27761 20.89 MB/sec warmup 80 sec latency 92.660 ms 2024-08-19T01:47:49.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 -- 172.21.15.64:0/1296808742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 msgr2=0x7f3160102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 --2- 172.21.15.64:0/1296808742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 0x7f3160102e40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f315c009930 tx=0x7f315c02a420 comp rx=0 tx=0).stop 2024-08-19T01:47:49.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 -- 172.21.15.64:0/1296808742 shutdown_connections 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 --2- 172.21.15.64:0/1296808742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3160107fd0 0x7f316010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 --2- 172.21.15.64:0/1296808742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 0x7f31601078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 --2- 172.21.15.64:0/1296808742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 0x7f3160102e40 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 -- 172.21.15.64:0/1296808742 >> 172.21.15.64:0/1296808742 conn(0x7f31600fc500 msgr2=0x7f31600fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 -- 172.21.15.64:0/1296808742 shutdown_connections 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 -- 172.21.15.64:0/1296808742 wait complete. 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 Processor -- start 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.565+0000 7f3168547640 1 -- start start 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3168547640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 0x7f31601982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3168547640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 0x7f3160198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3168547640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3160107fd0 0x7f316019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3168547640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3160111dd0 con 0x7f3160102a60 2024-08-19T01:47:49.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3168547640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3160111c50 con 0x7f3160103410 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3168547640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3160111f50 con 0x7f3160107fd0 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f31662e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 0x7f31601982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 0x7f3160198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 0x7f3160198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:39888/0 (socket says 172.21.15.64:39888) 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 -- 172.21.15.64:0/1697830185 learned_addr learned my addr 172.21.15.64:0/1697830185 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3166ae4640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3160107fd0 0x7f316019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 -- 172.21.15.64:0/1697830185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3160107fd0 msgr2=0x7f316019cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3160107fd0 0x7f316019cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 -- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 msgr2=0x7f31601982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 0x7f31601982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.566+0000 7f3165ae2640 1 -- 172.21.15.64:0/1697830185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316019d280 con 0x7f3160103410 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.567+0000 7f3166ae4640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3160107fd0 0x7f316019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.567+0000 7f3165ae2640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 0x7f3160198810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f315000e9f0 tx=0x7f315000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.567+0000 7f31577fe640 1 -- 172.21.15.64:0/1697830185 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315000ccc0 con 0x7f3160103410 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.567+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316019d570 con 0x7f3160103410 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.567+0000 7f31577fe640 1 -- 172.21.15.64:0/1697830185 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3150010910 con 0x7f3160103410 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.567+0000 7f31577fe640 1 -- 172.21.15.64:0/1697830185 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150018920 con 0x7f3160103410 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.567+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31601005f0 con 0x7f3160103410 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.570+0000 7f316a945740 1 Processor -- start 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.570+0000 7f316a945740 1 -- start start 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.570+0000 7f31577fe640 1 -- 172.21.15.64:0/1697830185 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3150010ac0 con 0x7f3160103410 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f316a945740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ff02c0 0x559272ff06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f316a945740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559272ffaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f316a945740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ffb4a0 0x559272ffd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f316a945740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559272f55810 con 0x559272ff0be0 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f316a945740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559272efcc70 con 0x559272ff02c0 2024-08-19T01:47:49.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f316a945740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559272f517d0 con 0x559272ffb4a0 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559272ffaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559272ffaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49946/0 (socket says 172.21.15.64:49946) 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 -- 172.21.15.64:0/386695215 learned_addr learned my addr 172.21.15.64:0/386695215 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f3165ae2640 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ffb4a0 0x559272ffd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f3166ae4640 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ff02c0 0x559272ff06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 -- 172.21.15.64:0/386695215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ffb4a0 msgr2=0x559272ffd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ffb4a0 0x559272ffd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 -- 172.21.15.64:0/386695215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ff02c0 msgr2=0x559272ff06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ff02c0 0x559272ff06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 -- 172.21.15.64:0/386695215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559272cc7ba0 con 0x559272ff0be0 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f3166ae4640 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ff02c0 0x559272ff06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31662e3640 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559272ffaf60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f315c02a940 tx=0x7f315c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7279002f0213d840 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:49.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.571+0000 7f31577fe640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f314c0a2cf0 0x7f314c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.572+0000 7f3166ae4640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f314c0a2cf0 0x7f314c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.572+0000 7f31577fe640 1 -- 172.21.15.64:0/1697830185 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f31500cfaf0 con 0x7f3160103410 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.572+0000 7f3154ff9640 1 -- 172.21.15.64:0/386695215 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c040070 con 0x559272ff0be0 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.572+0000 7f3154ff9640 1 -- 172.21.15.64:0/386695215 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c031870 con 0x559272ff0be0 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.572+0000 7f3154ff9640 1 -- 172.21.15.64:0/386695215 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c035040 con 0x559272ff0be0 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 -- 172.21.15.64:0/386695215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 msgr2=0x559272ffaf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f3166ae4640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f314c0a2cf0 0x7f314c0a51b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3158005fd0 tx=0x7f3158009500 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559272ffaf60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f315c02a940 tx=0x7f315c005bd0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 -- 172.21.15.64:0/386695215 shutdown_connections 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ffb4a0 0x559272ffd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559272ffaf60 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 --2- 172.21.15.64:0/386695215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ff02c0 0x559272ff06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 -- 172.21.15.64:0/386695215 >> 172.21.15.64:0/386695215 conn(0x559272fec850 msgr2=0x559272fecc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:49.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 -- 172.21.15.64:0/386695215 shutdown_connections 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.573+0000 7f316a945740 1 -- 172.21.15.64:0/386695215 wait complete. 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 -- start start 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 Processor -- start 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x559272ff8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559272ff89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559272ff8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559272ffdfb0 con 0x559272ff02c0 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559272f109a0 con 0x559272ffb4a0 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559272ffe130 con 0x559272ff0be0 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f316a945740 10 client.0 _ll_register_callbacks cb 0x559272efec90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f3165ae2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559272ff89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f3165ae2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559272ff89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:40168/0 (socket says 172.21.15.64:40168) 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.574+0000 7f3165ae2640 1 -- 172.21.15.64:0/3743764155 learned_addr learned my addr 172.21.15.64:0/3743764155 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:47:49.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.575+0000 7f3165ae2640 1 -- 172.21.15.64:0/3743764155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 msgr2=0x559272ff8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.575+0000 7f3165ae2640 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559272ff8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.575+0000 7f3165ae2640 1 -- 172.21.15.64:0/3743764155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 msgr2=0x559272ff8490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.575+0000 7f3165ae2640 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x559272ff8490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.575+0000 7f3165ae2640 1 -- 172.21.15.64:0/3743764155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559272f1c220 con 0x559272ff0be0 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.577+0000 7f3165ae2640 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559272ff89d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f315009b9b0 tx=0x7f3150009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f0694ced60f5f80 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.577+0000 7f3149ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.578+0000 7f3149ffb640 1 -- 172.21.15.64:0/3743764155 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315009b610 con 0x559272ff0be0 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.578+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f315c031870 con 0x559272ff0be0 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.578+0000 7f3149ffb640 1 -- 172.21.15.64:0/3743764155 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f315009b100 con 0x559272ff0be0 2024-08-19T01:47:49.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.578+0000 7f3149ffb640 1 -- 172.21.15.64:0/3743764155 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f315009bcb0 con 0x559272ff0be0 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.578+0000 7f316a945740 10 client.25492 fetch_fsmap learned FSMap version 30 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.578+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559273009c10 con 0x559272ff0be0 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.578+0000 7f3149ffb640 1 -- 172.21.15.64:0/3743764155 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c031870 con 0x559272ff0be0 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.579+0000 7f3149ffb640 1 -- 172.21.15.64:0/3743764155 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3150097b20 con 0x559272ff0be0 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.579+0000 7f316a945740 10 client.25492 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.579+0000 7f316a945740 10 client.25492 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.579+0000 7f316a945740 20 client.25492 populate_metadata read hostname 'smithi064' 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.579+0000 7f316a945740 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55927300b070 0x55927300d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.579+0000 7f316a945740 4 client.25492 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.579+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559272f62f40 con 0x55927300b070 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.580+0000 7f31662e3640 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55927300b070 0x55927300d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.581+0000 7f31662e3640 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55927300b070 0x55927300d510 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0bba2cc5a8e565 server_cookie=eb9ca74e44e90a5c in_seq=0 out_seq=0 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.581+0000 7f3149ffb640 10 client.25492 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:47:49.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f3149ffb640 1 -- 172.21.15.64:0/3743764155 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559272f62f40 con 0x55927300b070 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f3149ffb640 10 client.25492 handle_command_reply: tid=0 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 client.25492 shutdown 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 msgr2=0x559272ff89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559272ff89d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f315009b9b0 tx=0x7f3150009310 comp rx=0 tx=0).stop 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 shutdown_connections 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55927300b070 0x55927300d510 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559272ff8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559272ff89d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f3149ffb640 0 client.25492 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 --2- 172.21.15.64:0/3743764155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x559272ff8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.587+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 >> 172.21.15.64:0/3743764155 conn(0x559272fec850 msgr2=0x559272febb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.588+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 shutdown_connections 2024-08-19T01:47:49.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.588+0000 7f316a945740 1 -- 172.21.15.64:0/3743764155 wait complete. 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.588+0000 7f316a945740 20 client.25492 trim_cache size 0 max 16384 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 Processor -- start 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 -- start start 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff02c0 0x559273012de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559273013320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273003250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f31601b3a20 con 0x559272ff0be0 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31601b12f0 con 0x559272ffb4a0 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.607+0000 7f316a945740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31601af160 con 0x559272ff02c0 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559273013320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559273013320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49948/0 (socket says 172.21.15.64:49948) 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 -- 172.21.15.64:0/1156570335 learned_addr learned my addr 172.21.15.64:0/1156570335 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f31662e3640 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff02c0 0x559273012de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3166ae4640 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273003250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 -- 172.21.15.64:0/1156570335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff02c0 msgr2=0x559273012de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff02c0 0x559273012de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 -- 172.21.15.64:0/1156570335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 msgr2=0x559273003250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273003250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 -- 172.21.15.64:0/1156570335 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559272fed4e0 con 0x559272ff0be0 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3166ae4640 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273003250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f3165ae2640 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559273013320 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f31500a4340 tx=0x7f315009a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda07a196ee22e2a server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f314bfff640 1 -- 172.21.15.64:0/1156570335 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315009ba80 con 0x559272ff0be0 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f314bfff640 1 -- 172.21.15.64:0/1156570335 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315009af10 con 0x559272ff0be0 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.608+0000 7f314bfff640 1 -- 172.21.15.64:0/1156570335 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150097a00 con 0x559272ff0be0 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 -- 172.21.15.64:0/1156570335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 msgr2=0x559273013320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559273013320 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f31500a4340 tx=0x7f315009a1c0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 -- 172.21.15.64:0/1156570335 shutdown_connections 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273003250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff0be0 0x559273013320 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 --2- 172.21.15.64:0/1156570335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff02c0 0x559273012de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 -- 172.21.15.64:0/1156570335 >> 172.21.15.64:0/1156570335 conn(0x559272fec850 msgr2=0x559272febb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 -- 172.21.15.64:0/1156570335 shutdown_connections 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 -- 172.21.15.64:0/1156570335 wait complete. 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.609+0000 7f316a945740 1 -- start start 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 1 Processor -- start 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x5592730036c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559273000230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273000770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f31601af7b0 con 0x559272ff02c0 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31601b0c20 con 0x559272ffb4a0 2024-08-19T01:47:49.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31601afe80 con 0x559272ff0be0 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f316a945740 10 client.0 _ll_register_callbacks cb 0x7f315000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f31662e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x5592730036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f3165ae2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559273000230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f31662e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x5592730036c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49954/0 (socket says 172.21.15.64:49954) 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f31662e3640 1 -- 172.21.15.64:0/659779934 learned_addr learned my addr 172.21.15.64:0/659779934 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f3166ae4640 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273000770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f3165ae2640 1 -- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 msgr2=0x559273000770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f3165ae2640 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273000770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f3165ae2640 1 -- 172.21.15.64:0/659779934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 msgr2=0x5592730036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f3165ae2640 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x5592730036c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f3165ae2640 1 -- 172.21.15.64:0/659779934 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559273000cb0 con 0x559272ff0be0 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.610+0000 7f31662e3640 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x5592730036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:47:49.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f3165ae2640 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559273000230 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x559272ff98f0 tx=0x7f31500cfe80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85a07df240d99290 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f3149ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f3149ffb640 1 -- 172.21.15.64:0/659779934 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31500b8030 con 0x559272ff0be0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f315009af10 con 0x559272ff0be0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f3149ffb640 1 -- 172.21.15.64:0/659779934 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f31500b4e00 con 0x559272ff0be0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f3149ffb640 1 -- 172.21.15.64:0/659779934 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3150082050 con 0x559272ff0be0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 10 client.25498 fetch_fsmap learned FSMap version 30 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559273076ab0 con 0x559272ff0be0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f3149ffb640 1 -- 172.21.15.64:0/659779934 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315009af10 con 0x559272ff0be0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f3149ffb640 1 -- 172.21.15.64:0/659779934 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f315007cb90 con 0x559272ff0be0 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 10 client.25498 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 10 client.25498 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 20 client.25498 populate_metadata read hostname 'smithi064' 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559273077130 0x559273079550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 4 client.25498 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.65, "await_for": 45.0}] 2024-08-19T01:47:49.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.65, "await_for": 45.0}) -- 0x559272f5d290 con 0x559273077130 2024-08-19T01:47:49.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.611+0000 7f31662e3640 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559273077130 0x559273079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:47:49.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.612+0000 7f3149ffb640 10 client.25498 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:47:49.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:48.612+0000 7f31662e3640 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559273077130 0x559273079550 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30d3a6f06b9c0867 server_cookie=16ee0567ed92213c in_seq=0 out_seq=0 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f3149ffb640 1 -- 172.21.15.64:0/659779934 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559272f5d290 con 0x559273077130 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f3149ffb640 10 client.25498 handle_command_reply: tid=0 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 client.25498 shutdown 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 msgr2=0x559273000230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559273000230 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x559272ff98f0 tx=0x7f31500cfe80 comp rx=0 tx=0).stop 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 shutdown_connections 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x559273077130 0x559273079550 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559272ffb4a0 0x559273000770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559272ff0be0 0x559273000230 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f3149ffb640 0 client.25498 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 --2- 172.21.15.64:0/659779934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559272ff02c0 0x5592730036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 >> 172.21.15.64:0/659779934 conn(0x559272fec850 msgr2=0x559272ff1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 shutdown_connections 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 1 -- 172.21.15.64:0/659779934 wait complete. 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f316a945740 20 client.25498 trim_cache size 0 max 16384 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-08-19T01:47:49.270 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 50, 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "b4967359": { 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 50, 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 137.7, 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:47:49.271 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:47:49.272 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f314c0a2cf0 msgr2=0x7f314c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f3168547640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f314c0a2cf0 0x7f314c0a51b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3158005fd0 tx=0x7f3158009500 comp rx=0 tx=0).stop 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 msgr2=0x7f3160198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.268+0000 7f3168547640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 0x7f3160198810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f315000e9f0 tx=0x7f315000eeb0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 shutdown_connections 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f314c0a2cf0 0x7f314c0a51b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3160107fd0 0x7f316019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3160103410 0x7f3160198810 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 --2- 172.21.15.64:0/1697830185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160102a60 0x7f31601982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 >> 172.21.15.64:0/1697830185 conn(0x7f31600fc500 msgr2=0x7f3160105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:47:49.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 shutdown_connections 2024-08-19T01:47:49.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:47:49.269+0000 7f3168547640 1 -- 172.21.15.64:0/1697830185 wait complete. 2024-08-19T01:47:49.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 20.63 MB/sec warmup 81 sec latency 1033.055 ms 2024-08-19T01:47:50.134 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:49 smithi064 ceph-mon[18515]: pgmap v456: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 16 MiB/s wr, 687 op/s 2024-08-19T01:47:50.161 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b4967359', quiesce took 4.3 seconds. Will release after: 13.349999999999998 2024-08-19T01:47:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:49 smithi123 ceph-mon[21829]: pgmap v456: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 16 MiB/s wr, 687 op/s 2024-08-19T01:47:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:49 smithi161 ceph-mon[21757]: pgmap v456: 129 pgs: 129 active+clean; 916 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 16 MiB/s wr, 687 op/s 2024-08-19T01:47:50.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 20.38 MB/sec warmup 82 sec latency 2033.162 ms 2024-08-19T01:47:51.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 20.14 MB/sec warmup 83 sec latency 3033.246 ms 2024-08-19T01:47:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:51 smithi123 ceph-mon[21829]: pgmap v457: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 18 MiB/s wr, 842 op/s 2024-08-19T01:47:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:51 smithi161 ceph-mon[21757]: pgmap v457: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 18 MiB/s wr, 842 op/s 2024-08-19T01:47:52.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:51 smithi064 ceph-mon[18515]: pgmap v457: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 18 MiB/s wr, 842 op/s 2024-08-19T01:47:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 19.90 MB/sec warmup 84 sec latency 4033.284 ms 2024-08-19T01:47:53.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 19.66 MB/sec warmup 85 sec latency 5033.379 ms 2024-08-19T01:47:54.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:53 smithi123 ceph-mon[21829]: pgmap v458: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 427 op/s 2024-08-19T01:47:54.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:53 smithi161 ceph-mon[21757]: pgmap v458: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 427 op/s 2024-08-19T01:47:54.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:53 smithi064 ceph-mon[18515]: pgmap v458: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 427 op/s 2024-08-19T01:47:54.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 19.44 MB/sec warmup 86 sec latency 6033.508 ms 2024-08-19T01:47:55.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:54 smithi123 ceph-mon[21829]: pgmap v459: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 427 op/s 2024-08-19T01:47:55.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:54 smithi161 ceph-mon[21757]: pgmap v459: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 427 op/s 2024-08-19T01:47:55.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:54 smithi064 ceph-mon[18515]: pgmap v459: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 427 op/s 2024-08-19T01:47:55.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 19.21 MB/sec warmup 87 sec latency 7033.628 ms 2024-08-19T01:47:56.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 18.99 MB/sec warmup 88 sec latency 8033.720 ms 2024-08-19T01:47:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:57 smithi123 ceph-mon[21829]: pgmap v460: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.0 MiB/s wr, 437 op/s 2024-08-19T01:47:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:57 smithi161 ceph-mon[21757]: pgmap v460: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.0 MiB/s wr, 437 op/s 2024-08-19T01:47:57.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:57 smithi064 ceph-mon[18515]: pgmap v460: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.0 MiB/s wr, 437 op/s 2024-08-19T01:47:57.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 18.78 MB/sec warmup 89 sec latency 9033.821 ms 2024-08-19T01:47:58.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 18.57 MB/sec warmup 90 sec latency 10033.943 ms 2024-08-19T01:47:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:47:59 smithi123 ceph-mon[21829]: pgmap v461: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 164 op/s 2024-08-19T01:47:59.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:47:59 smithi161 ceph-mon[21757]: pgmap v461: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 164 op/s 2024-08-19T01:47:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:47:59 smithi064 ceph-mon[18515]: pgmap v461: 129 pgs: 129 active+clean; 932 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 164 op/s 2024-08-19T01:47:59.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 18.37 MB/sec warmup 91 sec latency 11034.049 ms 2024-08-19T01:48:00.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 18.17 MB/sec warmup 92 sec latency 12034.162 ms 2024-08-19T01:48:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:01 smithi123 ceph-mon[21829]: pgmap v462: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 170 op/s 2024-08-19T01:48:01.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:01 smithi161 ceph-mon[21757]: pgmap v462: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 170 op/s 2024-08-19T01:48:01.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:01 smithi064 ceph-mon[18515]: pgmap v462: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 170 op/s 2024-08-19T01:48:01.721 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 17.97 MB/sec warmup 93 sec latency 13034.268 ms 2024-08-19T01:48:02.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 17.78 MB/sec warmup 94 sec latency 14034.364 ms 2024-08-19T01:48:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:03 smithi123 ceph-mon[21829]: pgmap v463: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 606 KiB/s wr, 15 op/s 2024-08-19T01:48:03.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:03 smithi161 ceph-mon[21757]: pgmap v463: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 606 KiB/s wr, 15 op/s 2024-08-19T01:48:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:03 smithi064 ceph-mon[18515]: pgmap v463: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 606 KiB/s wr, 15 op/s 2024-08-19T01:48:03.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:48:03.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 17.59 MB/sec warmup 95 sec latency 15034.417 ms 2024-08-19T01:48:03.787 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:48:04.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4076253089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8108010 msgr2=0x7f5dc810e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:04.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4076253089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8108010 0x7f5dc810e980 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009930 tx=0x7f5dc002a420 comp rx=0 tx=0).stop 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4076253089 shutdown_connections 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4076253089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8108010 0x7f5dc810e980 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4076253089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dc8103490 0x7f5dc81078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4076253089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc8102ae0 0x7f5dc8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4076253089 >> 172.21.15.64:0/4076253089 conn(0x7f5dc80fc500 msgr2=0x7f5dc80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4076253089 shutdown_connections 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4076253089 wait complete. 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.293+0000 7f5dccd0e640 1 Processor -- start 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dccd0e640 1 -- start start 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dccd0e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8102ae0 0x7f5dc8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dccd0e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dc8103490 0x7f5dc81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dccd0e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc8108010 0x7f5dc819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:04.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dccd0e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8111de0 con 0x7f5dc8102ae0 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dccd0e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8111c60 con 0x7f5dc8103490 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dccd0e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8111f60 con 0x7f5dc8108010 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8102ae0 0x7f5dc8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8102ae0 0x7f5dc8198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39566/0 (socket says 172.21.15.64:39566) 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 -- 172.21.15.64:0/4125106463 learned_addr learned my addr 172.21.15.64:0/4125106463 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc5d9b640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dc8103490 0x7f5dc81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc6d9d640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc8108010 0x7f5dc819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 -- 172.21.15.64:0/4125106463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc8108010 msgr2=0x7f5dc819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc8108010 0x7f5dc819cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 -- 172.21.15.64:0/4125106463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dc8103490 msgr2=0x7f5dc81987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dc8103490 0x7f5dc81987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.294+0000 7f5dc659c640 1 -- 172.21.15.64:0/4125106463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc819d220 con 0x7f5dc8102ae0 2024-08-19T01:48:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.295+0000 7f5dc659c640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8102ae0 0x7f5dc8198270 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5db800eab0 tx=0x7f5db800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.295+0000 7f5dbf7fe640 1 -- 172.21.15.64:0/4125106463 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db800ce40 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.295+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc819d510 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.295+0000 7f5dbf7fe640 1 -- 172.21.15.64:0/4125106463 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db8011950 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.295+0000 7f5dbf7fe640 1 -- 172.21.15.64:0/4125106463 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8010ae0 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.295+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc81a4df0 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.296+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc8102ec0 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.297+0000 7f5dbf7fe640 1 -- 172.21.15.64:0/4125106463 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5db8011b00 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.297+0000 7f5dbf7fe640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5db40a2cf0 0x7f5db40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.298+0000 7f5dc5d9b640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5db40a2cf0 0x7f5db40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.298+0000 7f5dbf7fe640 1 -- 172.21.15.64:0/4125106463 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5db80cff50 con 0x7f5dc8102ae0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.298+0000 7f5dc5d9b640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5db40a2cf0 0x7f5db40a51b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80feb30 tx=0x7f5db000b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:04.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.302+0000 7f5dbf7fe640 1 -- 172.21.15.64:0/4125106463 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f5db8016030 con 0x7f5dc8102ae0 2024-08-19T01:48:04.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.513+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dc81995d0 con 0x7f5dc8102ae0 2024-08-19T01:48:04.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.514+0000 7f5dbf7fe640 1 -- 172.21.15.64:0/4125106463 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11924 (secure 0 0 0) 0x7f5db809c760 con 0x7f5dc8102ae0 2024-08-19T01:48:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:48:04.515 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"btime":"2024-08-19T01:46:31:794441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:46:31.560051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:48:04.515 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5db40a2cf0 msgr2=0x7f5db40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5db40a2cf0 0x7f5db40a51b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80feb30 tx=0x7f5db000b390 comp rx=0 tx=0).stop 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8102ae0 msgr2=0x7f5dc8198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8102ae0 0x7f5dc8198270 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5db800eab0 tx=0x7f5db800ef70 comp rx=0 tx=0).stop 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 shutdown_connections 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5db40a2cf0 0x7f5db40a51b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc8108010 0x7f5dc819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dc8103490 0x7f5dc81987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 --2- 172.21.15.64:0/4125106463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5dc8102ae0 0x7f5dc8198270 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 >> 172.21.15.64:0/4125106463 conn(0x7f5dc80fc500 msgr2=0x7f5dc8105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:04.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 shutdown_connections 2024-08-19T01:48:04.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:04.516+0000 7f5dccd0e640 1 -- 172.21.15.64:0/4125106463 wait complete. 2024-08-19T01:48:04.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 17.41 MB/sec warmup 96 sec latency 16034.507 ms 2024-08-19T01:48:05.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:05 smithi064 ceph-mon[18515]: pgmap v464: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 606 KiB/s wr, 15 op/s 2024-08-19T01:48:05.203 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:05 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4125106463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:05.204 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id b4967359 --release --await' 2024-08-19T01:48:05.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id b4967359 --release --await 2024-08-19T01:48:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:05 smithi123 ceph-mon[21829]: pgmap v464: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 606 KiB/s wr, 15 op/s 2024-08-19T01:48:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:05 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4125106463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:05.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:05 smithi161 ceph-mon[21757]: pgmap v464: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 606 KiB/s wr, 15 op/s 2024-08-19T01:48:05.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:05 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4125106463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:48:05.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 27773 17.23 MB/sec warmup 97 sec latency 17034.550 ms 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 -- 172.21.15.64:0/410456700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c101160 msgr2=0x7f5f7c10d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 --2- 172.21.15.64:0/410456700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c101160 0x7f5f7c10d690 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009cd0 tx=0x7f5f6c02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 -- 172.21.15.64:0/410456700 shutdown_connections 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 --2- 172.21.15.64:0/410456700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c101160 0x7f5f7c10d690 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 --2- 172.21.15.64:0/410456700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1007c0 0x7f5f7c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 --2- 172.21.15.64:0/410456700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f7c1067c0 0x7f5f7c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 -- 172.21.15.64:0/410456700 >> 172.21.15.64:0/410456700 conn(0x7f5f7c0fc500 msgr2=0x7f5f7c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:05.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 -- 172.21.15.64:0/410456700 shutdown_connections 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.959+0000 7f5f843c6640 1 -- 172.21.15.64:0/410456700 wait complete. 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 Processor -- start 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 -- start start 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c1007c0 0x7f5f7c196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f7c101160 0x7f5f7c197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1067c0 0x7f5f7c19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c110bc0 con 0x7f5f7c1007c0 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c110a40 con 0x7f5f7c1067c0 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f843c6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c110d40 con 0x7f5f7c101160 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c1007c0 0x7f5f7c196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c1007c0 0x7f5f7c196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39584/0 (socket says 172.21.15.64:39584) 2024-08-19T01:48:05.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 -- 172.21.15.64:0/633517939 learned_addr learned my addr 172.21.15.64:0/633517939 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:05.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f81961640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f7c101160 0x7f5f7c197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 -- 172.21.15.64:0/633517939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f7c101160 msgr2=0x7f5f7c197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:05.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f7c101160 0x7f5f7c197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 -- 172.21.15.64:0/633517939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1067c0 msgr2=0x7f5f7c19b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82963640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1067c0 0x7f5f7c19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1067c0 0x7f5f7c19b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82162640 1 -- 172.21.15.64:0/633517939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c19bf90 con 0x7f5f7c1007c0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.960+0000 7f5f82963640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1067c0 0x7f5f7c19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.961+0000 7f5f82162640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c1007c0 0x7f5f7c196fe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400d770 tx=0x7f5f7400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.961+0000 7f5f7b7fe640 1 -- 172.21.15.64:0/633517939 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f74017070 con 0x7f5f7c1007c0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.961+0000 7f5f7b7fe640 1 -- 172.21.15.64:0/633517939 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f74005500 con 0x7f5f7c1007c0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.961+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7c19c280 con 0x7f5f7c1007c0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.961+0000 7f5f7b7fe640 1 -- 172.21.15.64:0/633517939 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f74004e00 con 0x7f5f7c1007c0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.961+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7c1a3b60 con 0x7f5f7c1007c0 2024-08-19T01:48:05.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.963+0000 7f5f7b7fe640 1 -- 172.21.15.64:0/633517939 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5f74004fa0 con 0x7f5f7c1007c0 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.963+0000 7f5f867c4740 1 Processor -- start 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f867c4740 1 -- start start 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x56332978dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329798630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563329798b70 0x56332979af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f867c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5633296f3c60 con 0x56332978e2b0 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f867c4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56332968bb90 con 0x56332978d990 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f867c4740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5633296efc20 con 0x563329798b70 2024-08-19T01:48:05.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f7b7fe640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5f640a4f00 0x7f5f640a73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329798630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329798630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39592/0 (socket says 172.21.15.64:39592) 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 -- 172.21.15.64:0/1250419045 learned_addr learned my addr 172.21.15.64:0/1250419045 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82963640 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x56332978dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82963640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5f640a4f00 0x7f5f640a73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f7b7fe640 1 -- 172.21.15.64:0/633517939 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5f740cf330 con 0x7f5f7c1007c0 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f81961640 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563329798b70 0x56332979af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 -- 172.21.15.64:0/1250419045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563329798b70 msgr2=0x56332979af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563329798b70 0x56332979af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 -- 172.21.15.64:0/1250419045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 msgr2=0x56332978dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x56332978dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.964+0000 7f5f82162640 1 -- 172.21.15.64:0/1250419045 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563329403580 con 0x56332978e2b0 2024-08-19T01:48:05.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.965+0000 7f5f82963640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5f640a4f00 0x7f5f640a73c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009a60 tx=0x7f5f6c0095c0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.965+0000 7f5f82162640 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329798630 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5f740cc6d0 tx=0x7f5f7406e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5fde2700668ae2 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.965+0000 7f5f78dea640 1 -- 172.21.15.64:0/1250419045 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f74020070 con 0x56332978e2b0 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.965+0000 7f5f78dea640 1 -- 172.21.15.64:0/1250419045 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f740046c0 con 0x56332978e2b0 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.965+0000 7f5f78dea640 1 -- 172.21.15.64:0/1250419045 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f74005360 con 0x56332978e2b0 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.965+0000 7f5f867c4740 1 -- 172.21.15.64:0/1250419045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 msgr2=0x563329798630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.965+0000 7f5f867c4740 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329798630 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5f740cc6d0 tx=0x7f5f7406e4b0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- 172.21.15.64:0/1250419045 shutdown_connections 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563329798b70 0x56332979af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329798630 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 --2- 172.21.15.64:0/1250419045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x56332978dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- 172.21.15.64:0/1250419045 >> 172.21.15.64:0/1250419045 conn(0x563329789f20 msgr2=0x56332978a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- 172.21.15.64:0/1250419045 shutdown_connections 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- 172.21.15.64:0/1250419045 wait complete. 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- start start 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 Processor -- start 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329795ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978e2b0 0x563329796020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x563329796560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56332979b740 con 0x563329798b70 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5633296917c0 con 0x56332978e2b0 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56332979b8c0 con 0x56332978d990 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f867c4740 10 client.0 _ll_register_callbacks cb 0x56332969af50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f82963640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x563329796560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f82963640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x563329796560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39598/0 (socket says 172.21.15.64:39598) 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.966+0000 7f5f82963640 1 -- 172.21.15.64:0/3956521350 learned_addr learned my addr 172.21.15.64:0/3956521350 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f82162640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329795ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f81961640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978e2b0 0x563329796020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f82963640 1 -- 172.21.15.64:0/3956521350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 msgr2=0x563329795ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f82963640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329795ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f82963640 1 -- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978e2b0 msgr2=0x563329796020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f82963640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978e2b0 0x563329796020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f82963640 1 -- 172.21.15.64:0/3956521350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633294cd960 con 0x563329798b70 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f81961640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978e2b0 0x563329796020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f82963640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x563329796560 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c035000 tx=0x7f5f6c02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b18099841f6a74 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f61ffb640 1 -- 172.21.15.64:0/3956521350 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c051070 con 0x563329798b70 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f61ffb640 1 -- 172.21.15.64:0/3956521350 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5f6c004450 con 0x563329798b70 2024-08-19T01:48:05.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5633297011d0 con 0x563329798b70 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f61ffb640 1 -- 172.21.15.64:0/3956521350 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5f6c05d050 con 0x563329798b70 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f867c4740 10 client.16086 fetch_fsmap learned FSMap version 30 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.967+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5633297a7850 con 0x563329798b70 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f61ffb640 1 -- 172.21.15.64:0/3956521350 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5633297011d0 con 0x563329798b70 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f61ffb640 1 -- 172.21.15.64:0/3956521350 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5f6c030070 con 0x563329798b70 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f867c4740 10 client.16086 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f867c4740 10 client.16086 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f867c4740 20 client.16086 populate_metadata read hostname 'smithi064' 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f867c4740 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633297a8d20 0x5633297ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f867c4740 4 client.16086 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:48:05.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633296fb6e0 con 0x5633297a8d20 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f82162640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633297a8d20 0x5633297ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f61ffb640 10 client.16086 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.968+0000 7f5f82162640 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633297a8d20 0x5633297ab1a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0796854037edcac server_cookie=7a6bb6e201ba4925 in_seq=0 out_seq=0 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f61ffb640 1 -- 172.21.15.64:0/3956521350 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5633296fb6e0 con 0x5633297a8d20 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f61ffb640 10 client.16086 handle_command_reply: tid=0 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 client.16086 shutdown 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 msgr2=0x563329796560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x563329796560 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c035000 tx=0x7f5f6c02a8e0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 shutdown_connections 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x563329796560 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633297a8d20 0x5633297ab1a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978e2b0 0x563329796020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 --2- 172.21.15.64:0/3956521350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329795ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f61ffb640 0 client.16086 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:05.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 >> 172.21.15.64:0/3956521350 conn(0x563329789f20 msgr2=0x56332978fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:05.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 shutdown_connections 2024-08-19T01:48:05.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.969+0000 7f5f867c4740 1 -- 172.21.15.64:0/3956521350 wait complete. 2024-08-19T01:48:05.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.970+0000 7f5f867c4740 20 client.16086 trim_cache size 0 max 16384 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 Processor -- start 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 -- start start 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329790440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329797a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563329798b70 0x5633297b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1af580 con 0x56332978e2b0 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1ace50 con 0x563329798b70 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f867c4740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1aacc0 con 0x56332978d990 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329797a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329797a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39608/0 (socket says 172.21.15.64:39608) 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 -- 172.21.15.64:0/748597074 learned_addr learned my addr 172.21.15.64:0/748597074 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f82963640 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563329798b70 0x5633297b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f82162640 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329790440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 -- 172.21.15.64:0/748597074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 msgr2=0x563329790440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329790440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 -- 172.21.15.64:0/748597074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563329798b70 msgr2=0x5633297b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563329798b70 0x5633297b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.987+0000 7f5f81961640 1 -- 172.21.15.64:0/748597074 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56332978abb0 con 0x56332978e2b0 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f81961640 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329797a50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000ecf0 tx=0x7f5f70012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceacd4cdaede1063 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f63fff640 1 -- 172.21.15.64:0/748597074 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70004230 con 0x56332978e2b0 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f63fff640 1 -- 172.21.15.64:0/748597074 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f700043d0 con 0x56332978e2b0 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f63fff640 1 -- 172.21.15.64:0/748597074 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70003b00 con 0x56332978e2b0 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 -- 172.21.15.64:0/748597074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 msgr2=0x563329797a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329797a50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000ecf0 tx=0x7f5f70012820 comp rx=0 tx=0).stop 2024-08-19T01:48:05.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 -- 172.21.15.64:0/748597074 shutdown_connections 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563329798b70 0x5633297b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56332978e2b0 0x563329797a50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 --2- 172.21.15.64:0/748597074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978d990 0x563329790440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 -- 172.21.15.64:0/748597074 >> 172.21.15.64:0/748597074 conn(0x563329789f20 msgr2=0x563329797340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 -- 172.21.15.64:0/748597074 shutdown_connections 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.988+0000 7f5f867c4740 1 -- 172.21.15.64:0/748597074 wait complete. 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 -- start start 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 Processor -- start 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x5633297a1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978e2b0 0x56332979c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x56332979cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1ab310 con 0x563329798b70 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1ac780 con 0x56332978d990 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1ab9e0 con 0x56332978e2b0 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f867c4740 10 client.0 _ll_register_callbacks cb 0x7f5f6c0316d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x56332979cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x56332979cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39620/0 (socket says 172.21.15.64:39620) 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 -- 172.21.15.64:0/2109624778 learned_addr learned my addr 172.21.15.64:0/2109624778 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82162640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x5633297a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:05.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f81961640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978e2b0 0x56332979c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 -- 172.21.15.64:0/2109624778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978e2b0 msgr2=0x56332979c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978e2b0 0x56332979c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 -- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 msgr2=0x5633297a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x5633297a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82963640 1 -- 172.21.15.64:0/2109624778 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56332979d3b0 con 0x563329798b70 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.989+0000 7f5f82162640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x5633297a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f82963640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x56332979cb80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c035000 tx=0x7f5f6c02af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae61ac50636e930 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f61ffb640 1 -- 172.21.15.64:0/2109624778 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c066060 con 0x563329798b70 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f61ffb640 1 -- 172.21.15.64:0/2109624778 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5f6c04bd30 con 0x563329798b70 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f700043d0 con 0x563329798b70 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f61ffb640 1 -- 172.21.15.64:0/2109624778 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5f6c06d050 con 0x563329798b70 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 10 client.16092 fetch_fsmap learned FSMap version 30 2024-08-19T01:48:06.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563329813b20 con 0x563329798b70 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f61ffb640 1 -- 172.21.15.64:0/2109624778 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f700043d0 con 0x563329798b70 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f61ffb640 1 -- 172.21.15.64:0/2109624778 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5f6c030ad0 con 0x563329798b70 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 10 client.16092 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 10 client.16092 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 20 client.16092 populate_metadata read hostname 'smithi064' 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633298142a0 0x5633298166c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 4 client.16092 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b4967359", "release": true, "await": true}] 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.990+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b4967359", "release": true, "await": true}) -- 0x5633296785a0 con 0x5633298142a0 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.991+0000 7f5f82162640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633298142a0 0x5633298166c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.991+0000 7f5f82162640 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633298142a0 0x5633298166c0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a9d40d3c9a1a6a5 server_cookie=ae07822725c853f9 in_seq=0 out_seq=0 2024-08-19T01:48:06.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:05.991+0000 7f5f61ffb640 10 client.16092 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f61ffb640 1 -- 172.21.15.64:0/2109624778 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5633296785a0 con 0x5633298142a0 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f61ffb640 10 client.16092 handle_command_reply: tid=0 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 client.16092 shutdown 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 msgr2=0x56332979cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x56332979cb80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c035000 tx=0x7f5f6c02af20 comp rx=0 tx=0).stop 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 shutdown_connections 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563329798b70 0x56332979cb80 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56332978e2b0 0x56332979c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5633298142a0 0x5633298166c0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 --2- 172.21.15.64:0/2109624778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56332978d990 0x5633297a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f61ffb640 0 client.16092 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 >> 172.21.15.64:0/2109624778 conn(0x563329789f20 msgr2=0x5633297962c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 shutdown_connections 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.052+0000 7f5f867c4740 1 -- 172.21.15.64:0/2109624778 wait complete. 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f867c4740 20 client.16092 trim_cache size 0 max 16384 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 52, 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "b4967359": { 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 52, 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 137.7, 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5f640a4f00 msgr2=0x7f5f640a73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5f640a4f00 0x7f5f640a73c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009a60 tx=0x7f5f6c0095c0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c1007c0 msgr2=0x7f5f7c196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c1007c0 0x7f5f7c196fe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400d770 tx=0x7f5f7400dc30 comp rx=0 tx=0).stop 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 shutdown_connections 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5f640a4f00 0x7f5f640a73c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1067c0 0x7f5f7c19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f7c101160 0x7f5f7c197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 --2- 172.21.15.64:0/633517939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f7c1007c0 0x7f5f7c196fe0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:06.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 >> 172.21.15.64:0/633517939 conn(0x7f5f7c0fc500 msgr2=0x7f5f7c0fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:06.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 shutdown_connections 2024-08-19T01:48:06.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:06.053+0000 7f5f843c6640 1 -- 172.21.15.64:0/633517939 wait complete. 2024-08-19T01:48:06.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 28150 17.20 MB/sec warmup 98 sec latency 17329.359 ms 2024-08-19T01:48:06.723 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b4967359', total seconds elapsed: 20.9 2024-08-19T01:48:06.723 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.65 2024-08-19T01:48:06.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:48:07.036 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:48:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:07 smithi123 ceph-mon[21829]: pgmap v465: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 676 KiB/s wr, 35 op/s 2024-08-19T01:48:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:07 smithi161 ceph-mon[21757]: pgmap v465: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 676 KiB/s wr, 35 op/s 2024-08-19T01:48:07.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:07 smithi064 ceph-mon[18515]: pgmap v465: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 676 KiB/s wr, 35 op/s 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.576+0000 7f848cad6640 1 -- 172.21.15.64:0/2458067395 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8478004230 con 0x7f84881018b0 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.577+0000 7f848fd3c640 1 -- 172.21.15.64:0/2458067395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881018b0 msgr2=0x7f8488101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.577+0000 7f848fd3c640 1 --2- 172.21.15.64:0/2458067395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881018b0 0x7f8488101d10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8478009930 tx=0x7f847802a440 comp rx=0 tx=0).stop 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.577+0000 7f848fd3c640 1 -- 172.21.15.64:0/2458067395 shutdown_connections 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.577+0000 7f848fd3c640 1 --2- 172.21.15.64:0/2458067395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8488102250 0x7f848810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.577+0000 7f848fd3c640 1 --2- 172.21.15.64:0/2458067395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881018b0 0x7f8488101d10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.577+0000 7f848fd3c640 1 --2- 172.21.15.64:0/2458067395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84881078b0 0x7f8488107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.577+0000 7f848fd3c640 1 -- 172.21.15.64:0/2458067395 >> 172.21.15.64:0/2458067395 conn(0x7f84880fd610 msgr2=0x7f84880ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.578+0000 7f848fd3c640 1 -- 172.21.15.64:0/2458067395 shutdown_connections 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.578+0000 7f848fd3c640 1 -- 172.21.15.64:0/2458067395 wait complete. 2024-08-19T01:48:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.578+0000 7f848fd3c640 1 Processor -- start 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.578+0000 7f848fd3c640 1 -- start start 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848fd3c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84881018b0 0x7f8488198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848fd3c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8488102250 0x7f8488198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848fd3c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881078b0 0x7f848819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848fd3c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8488111d20 con 0x7f84881078b0 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848fd3c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8488111ba0 con 0x7f8488102250 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848fd3c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8488111ea0 con 0x7f84881018b0 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848dad8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84881018b0 0x7f8488198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848e2d9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881078b0 0x7f848819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848e2d9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881078b0 0x7f848819cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39630/0 (socket says 172.21.15.64:39630) 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.579+0000 7f848e2d9640 1 -- 172.21.15.64:0/3145785158 learned_addr learned my addr 172.21.15.64:0/3145785158 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.580+0000 7f848d2d7640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8488102250 0x7f8488198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.580+0000 7f848e2d9640 1 -- 172.21.15.64:0/3145785158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84881018b0 msgr2=0x7f8488198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:07.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.580+0000 7f848e2d9640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84881018b0 0x7f8488198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.580+0000 7f848e2d9640 1 -- 172.21.15.64:0/3145785158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8488102250 msgr2=0x7f8488198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.580+0000 7f848e2d9640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8488102250 0x7f8488198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.580+0000 7f848e2d9640 1 -- 172.21.15.64:0/3145785158 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848819d200 con 0x7f84881078b0 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.580+0000 7f848e2d9640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881078b0 0x7f848819cb20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f848000b970 tx=0x7f848000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.581+0000 7f847effd640 1 -- 172.21.15.64:0/3145785158 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8480019070 con 0x7f84881078b0 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.581+0000 7f847effd640 1 -- 172.21.15.64:0/3145785158 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f848000cb40 con 0x7f84881078b0 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.581+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848819d4f0 con 0x7f84881078b0 2024-08-19T01:48:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.583+0000 7f847effd640 1 -- 172.21.15.64:0/3145785158 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8480007650 con 0x7f84881078b0 2024-08-19T01:48:07.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.583+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8488074150 con 0x7f84881078b0 2024-08-19T01:48:07.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.585+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8488102ff0 con 0x7f84881078b0 2024-08-19T01:48:07.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.586+0000 7f847effd640 1 -- 172.21.15.64:0/3145785158 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f848001c070 con 0x7f84881078b0 2024-08-19T01:48:07.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.587+0000 7f847effd640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f84740a2e40 0x7f84740a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:07.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.587+0000 7f847effd640 1 -- 172.21.15.64:0/3145785158 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7f84800cec70 con 0x7f84881078b0 2024-08-19T01:48:07.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.588+0000 7f848dad8640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f84740a2e40 0x7f84740a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:07.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.591+0000 7f848dad8640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f84740a2e40 0x7f84740a5300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8484009aa0 tx=0x7f8484005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.595+0000 7f847effd640 1 -- 172.21.15.64:0/3145785158 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f848009aef0 con 0x7f84881078b0 2024-08-19T01:48:07.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 28690 17.31 MB/sec warmup 99 sec latency 173.947 ms 2024-08-19T01:48:07.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.890+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8488109e40 con 0x7f84881078b0 2024-08-19T01:48:07.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.891+0000 7f847effd640 1 -- 172.21.15.64:0/3145785158 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11924 (secure 0 0 0) 0x7f848009f1f0 con 0x7f84881078b0 2024-08-19T01:48:07.892 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:48:07.892 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"btime":"2024-08-19T01:46:31:794441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:46:31.560051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:48:07.893 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f84740a2e40 msgr2=0x7f84740a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f84740a2e40 0x7f84740a5300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8484009aa0 tx=0x7f8484005f90 comp rx=0 tx=0).stop 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881078b0 msgr2=0x7f848819cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881078b0 0x7f848819cb20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f848000b970 tx=0x7f848000be30 comp rx=0 tx=0).stop 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 shutdown_connections 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f84740a2e40 0x7f84740a5300 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84881078b0 0x7f848819cb20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8488102250 0x7f8488198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 --2- 172.21.15.64:0/3145785158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84881018b0 0x7f8488198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:07.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 >> 172.21.15.64:0/3145785158 conn(0x7f84880fd610 msgr2=0x7f84880fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:07.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 shutdown_connections 2024-08-19T01:48:07.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:07.894+0000 7f848fd3c640 1 -- 172.21.15.64:0/3145785158 wait complete. 2024-08-19T01:48:08.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:08 smithi064 ceph-mon[18515]: osdmap e111: 12 total, 12 up, 12 in 2024-08-19T01:48:08.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:08 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3145785158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:08.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:08 smithi123 ceph-mon[21829]: osdmap e111: 12 total, 12 up, 12 in 2024-08-19T01:48:08.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:08 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3145785158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:08.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:08 smithi161 ceph-mon[21757]: osdmap e111: 12 total, 12 up, 12 in 2024-08-19T01:48:08.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:08 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3145785158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:08.587 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 137.65 --await-for 45.0' 2024-08-19T01:48:08.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 137.65 --await-for 45.0 2024-08-19T01:48:08.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29330 17.56 MB/sec warmup 100 sec latency 93.754 ms 2024-08-19T01:48:08.882 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:48:09.368 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:09 smithi064 ceph-mon[18515]: pgmap v467: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 118 KiB/s wr, 31 op/s 2024-08-19T01:48:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:09 smithi123 ceph-mon[21829]: pgmap v467: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 118 KiB/s wr, 31 op/s 2024-08-19T01:48:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:09 smithi161 ceph-mon[21757]: pgmap v467: 129 pgs: 129 active+clean; 933 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 118 KiB/s wr, 31 op/s 2024-08-19T01:48:09.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/2202048730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc341034b0 msgr2=0x7fcc341078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/2202048730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc341034b0 0x7fcc341078f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009930 tx=0x7fcc2802a420 comp rx=0 tx=0).stop 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/2202048730 shutdown_connections 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/2202048730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc34108030 0x7fcc3410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/2202048730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc341034b0 0x7fcc341078f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/2202048730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc34102b00 0x7fcc34102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/2202048730 >> 172.21.15.64:0/2202048730 conn(0x7fcc340fc500 msgr2=0x7fcc340fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/2202048730 shutdown_connections 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.513+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/2202048730 wait complete. 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 Processor -- start 2024-08-19T01:48:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 -- start start 2024-08-19T01:48:09.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc34102b00 0x7fcc341a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc341034b0 0x7fcc341a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc34108030 0x7fcc3419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3410fbf0 con 0x7fcc34108030 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3410fa70 con 0x7fcc341034b0 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc3a0f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3410fd70 con 0x7fcc34102b00 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc33fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc34108030 0x7fcc3419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc32ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc341034b0 0x7fcc341a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc33fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc34108030 0x7fcc3419e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33970/0 (socket says 172.21.15.64:33970) 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc33fff640 1 -- 172.21.15.64:0/3456596312 learned_addr learned my addr 172.21.15.64:0/3456596312 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.514+0000 7fcc337fe640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc34102b00 0x7fcc341a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc337fe640 1 -- 172.21.15.64:0/3456596312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc341034b0 msgr2=0x7fcc341a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc337fe640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc341034b0 0x7fcc341a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc337fe640 1 -- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc34108030 msgr2=0x7fcc3419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc337fe640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc34108030 0x7fcc3419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc337fe640 1 -- 172.21.15.64:0/3456596312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3419ef80 con 0x7fcc34102b00 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc341034b0 0x7fcc341a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:48:09.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc337fe640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc34102b00 0x7fcc341a4640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400bb20 tx=0x7fcc240091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:09.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc30ff9640 1 -- 172.21.15.64:0/3456596312 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24009af0 con 0x7fcc34102b00 2024-08-19T01:48:09.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3419f270 con 0x7fcc34102b00 2024-08-19T01:48:09.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc30ff9640 1 -- 172.21.15.64:0/3456596312 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc24009c90 con 0x7fcc34102b00 2024-08-19T01:48:09.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc30ff9640 1 -- 172.21.15.64:0/3456596312 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24015620 con 0x7fcc34102b00 2024-08-19T01:48:09.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.515+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc341ab5c0 con 0x7fcc34102b00 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.518+0000 7fcc30ff9640 1 -- 172.21.15.64:0/3456596312 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcc24016420 con 0x7fcc34102b00 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.518+0000 7fcc3c4f2740 1 Processor -- start 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.518+0000 7fcc3c4f2740 1 -- start start 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.518+0000 7fcc30ff9640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcc1c0a2cf0 0x7fcc1c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.518+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceb9f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9fbe0 0x5609ceba9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cebac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5609ceb04810 con 0x5609ceb9f2c0 2024-08-19T01:48:09.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5609ceaabc70 con 0x5609ceb9fbe0 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5609ceb007d0 con 0x5609cebaa4a0 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc30ff9640 1 -- 172.21.15.64:0/3456596312 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fcc240cf670 con 0x7fcc34102b00 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceb9f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceb9f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33980/0 (socket says 172.21.15.64:33980) 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 -- 172.21.15.64:0/118022037 learned_addr learned my addr 172.21.15.64:0/118022037 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9fbe0 0x5609ceba9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc33fff640 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cebac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 -- 172.21.15.64:0/118022037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 msgr2=0x5609cebac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cebac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 -- 172.21.15.64:0/118022037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9fbe0 msgr2=0x5609ceba9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9fbe0 0x5609ceba9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 -- 172.21.15.64:0/118022037 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609ce876ba0 con 0x5609ceb9f2c0 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc33fff640 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cebac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc33fff640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcc1c0a2cf0 0x7fcc1c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.519+0000 7fcc337fe640 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceb9f6a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fcc24097b90 tx=0x7fcc2409b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d49d0207cfcbc2 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.520+0000 7fcc19ffb640 1 -- 172.21.15.64:0/118022037 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24009e40 con 0x5609ceb9f2c0 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.520+0000 7fcc19ffb640 1 -- 172.21.15.64:0/118022037 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc2409be30 con 0x5609ceb9f2c0 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.520+0000 7fcc19ffb640 1 -- 172.21.15.64:0/118022037 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc240ccbe0 con 0x5609ceb9f2c0 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.520+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/118022037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 msgr2=0x5609ceb9f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.520+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceb9f6a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fcc24097b90 tx=0x7fcc2409b330 comp rx=0 tx=0).stop 2024-08-19T01:48:09.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.520+0000 7fcc33fff640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcc1c0a2cf0 0x7fcc1c0a51b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0094d0 tx=0x7fcc2c009500 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/118022037 shutdown_connections 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cebac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9fbe0 0x5609ceba9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/118022037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceb9f6a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/118022037 >> 172.21.15.64:0/118022037 conn(0x5609ceb9b850 msgr2=0x5609ceb9bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/118022037 shutdown_connections 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/118022037 wait complete. 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 -- start start 2024-08-19T01:48:09.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 Processor -- start 2024-08-19T01:48:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.521+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9f2c0 0x5609ceba7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609ceba79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609ceba7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5609cebacfb0 con 0x5609ceb9fbe0 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5609ceabf9a0 con 0x5609cebaa4a0 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5609cebad130 con 0x5609ceb9f2c0 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc3c4f2740 10 client.0 _ll_register_callbacks cb 0x5609ceaadc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9f2c0 0x5609ceba7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609ceba79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9f2c0 0x5609ceba7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:35520/0 (socket says 172.21.15.64:35520) 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609ceba79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33988/0 (socket says 172.21.15.64:33988) 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc337fe640 1 -- 172.21.15.64:0/1679951747 learned_addr learned my addr 172.21.15.64:0/1679951747 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc33fff640 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609ceba7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 -- 172.21.15.64:0/1679951747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9f2c0 msgr2=0x5609ceba7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9f2c0 0x5609ceba7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 -- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 msgr2=0x5609ceba7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609ceba7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 -- 172.21.15.64:0/1679951747 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609ceacb220 con 0x5609ceb9fbe0 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc337fe640 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9f2c0 0x5609ceba7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609ceba79a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fcc2802a940 tx=0x7fcc280317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3478448a35d768a8 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc12ffd640 1 -- 172.21.15.64:0/1679951747 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28035040 con 0x5609ceb9fbe0 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc12ffd640 1 -- 172.21.15.64:0/1679951747 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fcc28004040 con 0x5609ceb9fbe0 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.522+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc2409be30 con 0x5609ceb9fbe0 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc12ffd640 1 -- 172.21.15.64:0/1679951747 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcc28004240 con 0x5609ceb9fbe0 2024-08-19T01:48:09.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 10 client.16110 fetch_fsmap learned FSMap version 30 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5609cebb8c00 con 0x5609ceb9fbe0 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc12ffd640 1 -- 172.21.15.64:0/1679951747 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc2409be30 con 0x5609ceb9fbe0 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc12ffd640 1 -- 172.21.15.64:0/1679951747 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcc28004aa0 con 0x5609ceb9fbe0 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 10 client.16110 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 10 client.16110 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 20 client.16110 populate_metadata read hostname 'smithi064' 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cebba0c0 0x5609cebbc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 4 client.16110 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609ceb11f40 con 0x5609cebba0c0 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.523+0000 7fcc337fe640 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cebba0c0 0x5609cebbc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.524+0000 7fcc12ffd640 10 client.16110 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.524+0000 7fcc337fe640 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cebba0c0 0x5609cebbc510 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b450c84475d7b773 server_cookie=800e95cd7ae18327 in_seq=0 out_seq=0 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.525+0000 7fcc12ffd640 1 -- 172.21.15.64:0/1679951747 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5609ceb11f40 con 0x5609cebba0c0 2024-08-19T01:48:09.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.525+0000 7fcc12ffd640 10 client.16110 handle_command_reply: tid=0 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.525+0000 7fcc3c4f2740 1 client.16110 shutdown 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 msgr2=0x5609ceba79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609ceba79a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fcc2802a940 tx=0x7fcc280317c0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 shutdown_connections 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cebba0c0 0x5609cebbc510 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609ceba7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609ceba79a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/1679951747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9f2c0 0x5609ceba7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc12ffd640 0 client.16110 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 >> 172.21.15.64:0/1679951747 conn(0x5609ceb9b850 msgr2=0x5609ceb9ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 shutdown_connections 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/1679951747 wait complete. 2024-08-19T01:48:09.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.526+0000 7fcc3c4f2740 20 client.16110 trim_cache size 0 max 16384 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 Processor -- start 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 -- start start 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceba1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9fbe0 0x5609cebc1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609cebc2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b7fb0 con 0x5609ceb9f2c0 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b5880 con 0x5609cebaa4a0 2024-08-19T01:48:09.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b36f0 con 0x5609ceb9fbe0 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceba1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceba1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33992/0 (socket says 172.21.15.64:33992) 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc337fe640 1 -- 172.21.15.64:0/3763218994 learned_addr learned my addr 172.21.15.64:0/3763218994 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.550+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9fbe0 0x5609cebc1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc33fff640 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609cebc2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc337fe640 1 -- 172.21.15.64:0/3763218994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9fbe0 msgr2=0x5609cebc1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc337fe640 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9fbe0 0x5609cebc1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc337fe640 1 -- 172.21.15.64:0/3763218994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 msgr2=0x5609cebc2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc337fe640 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609cebc2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc337fe640 1 -- 172.21.15.64:0/3763218994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609ceb9b680 con 0x5609ceb9f2c0 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc337fe640 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceba1d20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fcc2409b430 tx=0x7fcc24086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d8896f6cec7e22 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:09.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc197fa640 1 -- 172.21.15.64:0/3763218994 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc240b4070 con 0x5609ceb9f2c0 2024-08-19T01:48:09.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc197fa640 1 -- 172.21.15.64:0/3763218994 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc2409be30 con 0x5609ceb9f2c0 2024-08-19T01:48:09.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.551+0000 7fcc197fa640 1 -- 172.21.15.64:0/3763218994 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24097da0 con 0x5609ceb9f2c0 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/3763218994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 msgr2=0x5609ceba1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceba1d20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fcc2409b430 tx=0x7fcc24086040 comp rx=0 tx=0).stop 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/3763218994 shutdown_connections 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609cebaa4a0 0x5609cebc2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609ceb9fbe0 0x5609cebc1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/3763218994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9f2c0 0x5609ceba1d20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/3763218994 >> 172.21.15.64:0/3763218994 conn(0x5609ceb9b850 msgr2=0x5609ceb9ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/3763218994 shutdown_connections 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.552+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/3763218994 wait complete. 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 -- start start 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 Processor -- start 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9f2c0 0x5609cebae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609cebaea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cec23df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b3d40 con 0x5609ceb9fbe0 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b51b0 con 0x5609ceb9f2c0 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.553+0000 7fcc3c4f2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b4410 con 0x5609cebaa4a0 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc3c4f2740 10 client.0 _ll_register_callbacks cb 0x7fcc28009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609cebaea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9f2c0 0x5609cebae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609cebaea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34000/0 (socket says 172.21.15.64:34000) 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 -- 172.21.15.64:0/2544817191 learned_addr learned my addr 172.21.15.64:0/2544817191 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc33fff640 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cec23df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 -- 172.21.15.64:0/2544817191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 msgr2=0x5609cec23df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cec23df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 -- 172.21.15.64:0/2544817191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9f2c0 msgr2=0x5609cebae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9f2c0 0x5609cebae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 -- 172.21.15.64:0/2544817191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609ceb9c900 con 0x5609ceb9fbe0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc33fff640 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cec23df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc32ffd640 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609cebaea10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcc28004ef0 tx=0x7fcc28004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11e8abd33d58417 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc12ffd640 1 -- 172.21.15.64:0/2544817191 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28035070 con 0x5609ceb9fbe0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.554+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc2409be30 con 0x5609ceb9fbe0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc12ffd640 1 -- 172.21.15.64:0/2544817191 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7fcc28004970 con 0x5609ceb9fbe0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc12ffd640 1 -- 172.21.15.64:0/2544817191 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcc28004b70 con 0x5609ceb9fbe0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc3c4f2740 10 client.16116 fetch_fsmap learned FSMap version 30 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5609cec252e0 con 0x5609ceb9fbe0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc12ffd640 1 -- 172.21.15.64:0/2544817191 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc2409be30 con 0x5609ceb9fbe0 2024-08-19T01:48:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc12ffd640 1 -- 172.21.15.64:0/2544817191 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcc28005350 con 0x5609ceb9fbe0 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc3c4f2740 10 client.16116 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc3c4f2740 10 client.16116 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc3c4f2740 20 client.16116 populate_metadata read hostname 'smithi064' 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.555+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cec25bb0 0x5609cec27fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.556+0000 7fcc3c4f2740 4 client.16116 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.65, "await_for": 45.0}] 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.556+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.65, "await_for": 45.0}) -- 0x5609ceb0c290 con 0x5609cec25bb0 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.556+0000 7fcc337fe640 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cec25bb0 0x5609cec27fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.557+0000 7fcc12ffd640 10 client.16116 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:09.557+0000 7fcc337fe640 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cec25bb0 0x5609cec27fd0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f178b5a591dbfc2a server_cookie=c1d345da23d41652 in_seq=0 out_seq=0 2024-08-19T01:48:09.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29756 17.60 MB/sec warmup 101 sec latency 146.978 ms 2024-08-19T01:48:10.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.313+0000 7fcc12ffd640 1 -- 172.21.15.64:0/2544817191 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5609ceb0c290 con 0x5609cec25bb0 2024-08-19T01:48:10.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.313+0000 7fcc12ffd640 10 client.16116 handle_command_reply: tid=0 2024-08-19T01:48:10.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.313+0000 7fcc3c4f2740 1 client.16116 shutdown 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 msgr2=0x5609cebaea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609cebaea10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcc28004ef0 tx=0x7fcc28004610 comp rx=0 tx=0).stop 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 shutdown_connections 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5609cebaa4a0 0x5609cec23df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609ceb9fbe0 0x5609cebaea10 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5609cec25bb0 0x5609cec27fd0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 --2- 172.21.15.64:0/2544817191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5609ceb9f2c0 0x5609cebae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc12ffd640 0 client.16116 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 >> 172.21.15.64:0/2544817191 conn(0x5609ceb9b850 msgr2=0x5609ceb9ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 shutdown_connections 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 1 -- 172.21.15.64:0/2544817191 wait complete. 2024-08-19T01:48:10.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.314+0000 7fcc3c4f2740 20 client.16116 trim_cache size 0 max 16384 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 54, 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "3ee1542d": { 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 54, 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:48:10.317 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 137.7, 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:10.318 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:48:10.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.315+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcc1c0a2cf0 msgr2=0x7fcc1c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.315+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcc1c0a2cf0 0x7fcc1c0a51b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0094d0 tx=0x7fcc2c009500 comp rx=0 tx=0).stop 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.315+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc34102b00 msgr2=0x7fcc341a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.315+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc34102b00 0x7fcc341a4640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400bb20 tx=0x7fcc240091e0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 shutdown_connections 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fcc1c0a2cf0 0x7fcc1c0a51b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc34108030 0x7fcc3419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc341034b0 0x7fcc341a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 --2- 172.21.15.64:0/3456596312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc34102b00 0x7fcc341a4640 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 >> 172.21.15.64:0/3456596312 conn(0x7fcc340fc500 msgr2=0x7fcc34105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 shutdown_connections 2024-08-19T01:48:10.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:10.316+0000 7fcc3a0f4640 1 -- 172.21.15.64:0/3456596312 wait complete. 2024-08-19T01:48:10.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 17.43 MB/sec warmup 102 sec latency 1131.324 ms 2024-08-19T01:48:11.009 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3ee1542d', quiesce took 4.3 seconds. Will release after: 13.349999999999998 2024-08-19T01:48:11.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:11 smithi064 ceph-mon[18515]: pgmap v468: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-08-19T01:48:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:11 smithi123 ceph-mon[21829]: pgmap v468: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-08-19T01:48:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:11 smithi161 ceph-mon[21757]: pgmap v468: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-08-19T01:48:11.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 17.26 MB/sec warmup 103 sec latency 2131.373 ms 2024-08-19T01:48:12.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 17.10 MB/sec warmup 104 sec latency 3131.417 ms 2024-08-19T01:48:13.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:13 smithi123 ceph-mon[21829]: pgmap v469: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-08-19T01:48:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:13 smithi161 ceph-mon[21757]: pgmap v469: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-08-19T01:48:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:13 smithi064 ceph-mon[18515]: pgmap v469: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-08-19T01:48:13.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 16.93 MB/sec warmup 105 sec latency 4131.477 ms 2024-08-19T01:48:14.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 16.77 MB/sec warmup 106 sec latency 5131.579 ms 2024-08-19T01:48:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:14 smithi123 ceph-mon[21829]: osdmap e112: 12 total, 12 up, 12 in 2024-08-19T01:48:15.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:14 smithi161 ceph-mon[21757]: osdmap e112: 12 total, 12 up, 12 in 2024-08-19T01:48:15.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:14 smithi064 ceph-mon[18515]: osdmap e112: 12 total, 12 up, 12 in 2024-08-19T01:48:15.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 16.62 MB/sec warmup 107 sec latency 6131.708 ms 2024-08-19T01:48:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:15 smithi123 ceph-mon[21829]: pgmap v471: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.5 MiB/s wr, 249 op/s 2024-08-19T01:48:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:15 smithi161 ceph-mon[21757]: pgmap v471: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.5 MiB/s wr, 249 op/s 2024-08-19T01:48:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:15 smithi064 ceph-mon[18515]: pgmap v471: 129 pgs: 129 active+clean; 947 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.5 MiB/s wr, 249 op/s 2024-08-19T01:48:16.722 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 16.46 MB/sec warmup 108 sec latency 7131.794 ms 2024-08-19T01:48:17.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 16.31 MB/sec warmup 109 sec latency 8131.834 ms 2024-08-19T01:48:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:17 smithi123 ceph-mon[21829]: pgmap v472: 129 pgs: 129 active+clean; 970 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 304 op/s 2024-08-19T01:48:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:17 smithi161 ceph-mon[21757]: pgmap v472: 129 pgs: 129 active+clean; 970 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 304 op/s 2024-08-19T01:48:18.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:17 smithi064 ceph-mon[18515]: pgmap v472: 129 pgs: 129 active+clean; 970 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.9 MiB/s wr, 304 op/s 2024-08-19T01:48:18.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 16.16 MB/sec warmup 110 sec latency 9131.874 ms 2024-08-19T01:48:19.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 16.02 MB/sec warmup 111 sec latency 10131.983 ms 2024-08-19T01:48:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:19 smithi123 ceph-mon[21829]: pgmap v473: 129 pgs: 129 active+clean; 970 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 271 op/s 2024-08-19T01:48:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:19 smithi161 ceph-mon[21757]: pgmap v473: 129 pgs: 129 active+clean; 970 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 271 op/s 2024-08-19T01:48:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:19 smithi064 ceph-mon[18515]: pgmap v473: 129 pgs: 129 active+clean; 970 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 271 op/s 2024-08-19T01:48:20.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 15.87 MB/sec warmup 112 sec latency 11132.080 ms 2024-08-19T01:48:21.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 15.73 MB/sec warmup 113 sec latency 12132.191 ms 2024-08-19T01:48:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:21 smithi123 ceph-mon[21829]: pgmap v474: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 4.9 MiB/s wr, 175 op/s 2024-08-19T01:48:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:21 smithi161 ceph-mon[21757]: pgmap v474: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 4.9 MiB/s wr, 175 op/s 2024-08-19T01:48:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:21 smithi064 ceph-mon[18515]: pgmap v474: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 4.9 MiB/s wr, 175 op/s 2024-08-19T01:48:22.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 15.60 MB/sec warmup 114 sec latency 13132.315 ms 2024-08-19T01:48:23.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 15.46 MB/sec warmup 115 sec latency 14132.399 ms 2024-08-19T01:48:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:23 smithi123 ceph-mon[21829]: pgmap v475: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 4.9 MiB/s wr, 175 op/s 2024-08-19T01:48:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:23 smithi161 ceph-mon[21757]: pgmap v475: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 4.9 MiB/s wr, 175 op/s 2024-08-19T01:48:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:23 smithi064 ceph-mon[18515]: pgmap v475: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 4.9 MiB/s wr, 175 op/s 2024-08-19T01:48:24.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:48:24.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 15.33 MB/sec warmup 116 sec latency 15132.481 ms 2024-08-19T01:48:25.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 -- 172.21.15.64:0/1084471152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85641022a0 msgr2=0x7f85641026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:25.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 --2- 172.21.15.64:0/1084471152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85641022a0 0x7f85641026e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8554009900 tx=0x7f855402a480 comp rx=0 tx=0).stop 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 -- 172.21.15.64:0/1084471152 shutdown_connections 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 --2- 172.21.15.64:0/1084471152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8564102c20 0x7f856410d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 --2- 172.21.15.64:0/1084471152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85641022a0 0x7f85641026e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 --2- 172.21.15.64:0/1084471152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8564100b50 0x7f8564100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 -- 172.21.15.64:0/1084471152 >> 172.21.15.64:0/1084471152 conn(0x7f85640fc500 msgr2=0x7f85640fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 -- 172.21.15.64:0/1084471152 shutdown_connections 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.135+0000 7f856ba0f640 1 -- 172.21.15.64:0/1084471152 wait complete. 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 Processor -- start 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 -- start start 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8564100b50 0x7f8564196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85641022a0 0x7f85641972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8564102c20 0x7f85641977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:25.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8564110cb0 con 0x7f8564100b50 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8564110b30 con 0x7f8564102c20 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f856ba0f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8564110e30 con 0x7f85641022a0 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f85697ab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8564100b50 0x7f8564196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f85697ab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8564100b50 0x7f8564196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40752/0 (socket says 172.21.15.64:40752) 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f85697ab640 1 -- 172.21.15.64:0/3617536964 learned_addr learned my addr 172.21.15.64:0/3617536964 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f8569fac640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8564102c20 0x7f85641977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.136+0000 7f8568faa640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85641022a0 0x7f85641972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f85697ab640 1 -- 172.21.15.64:0/3617536964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85641022a0 msgr2=0x7f85641972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f85697ab640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85641022a0 0x7f85641972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f85697ab640 1 -- 172.21.15.64:0/3617536964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8564102c20 msgr2=0x7f85641977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f85697ab640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8564102c20 0x7f85641977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f85697ab640 1 -- 172.21.15.64:0/3617536964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85641a65e0 con 0x7f8564100b50 2024-08-19T01:48:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f85697ab640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8564100b50 0x7f8564196d70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f856000c920 tx=0x7f856000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f855a7fc640 1 -- 172.21.15.64:0/3617536964 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560007c70 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85641a68d0 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f855a7fc640 1 -- 172.21.15.64:0/3617536964 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8560007e10 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f855a7fc640 1 -- 172.21.15.64:0/3617536964 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560018890 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.137+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85641a6e60 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.138+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8564100f30 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.139+0000 7f855a7fc640 1 -- 172.21.15.64:0/3617536964 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8560018a50 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.140+0000 7f855a7fc640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85500a2cf0 0x7f85500a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.140+0000 7f855a7fc640 1 -- 172.21.15.64:0/3617536964 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7f85600cfa80 con 0x7f8564100b50 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.140+0000 7f8568faa640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85500a2cf0 0x7f85500a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.141+0000 7f8568faa640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85500a2cf0 0x7f85500a51b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8564198ed0 tx=0x7f8554005a20 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:25.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.148+0000 7f855a7fc640 1 -- 172.21.15.64:0/3617536964 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f8560014030 con 0x7f8564100b50 2024-08-19T01:48:25.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.393+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85641a7150 con 0x7f8564100b50 2024-08-19T01:48:25.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.398+0000 7f855a7fc640 1 -- 172.21.15.64:0/3617536964 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11924 (secure 0 0 0) 0x7f856009c290 con 0x7f8564100b50 2024-08-19T01:48:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:48:25.401 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"btime":"2024-08-19T01:46:31:794441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:46:31.560051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:48:25.401 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-08-19T01:48:25.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.403+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85500a2cf0 msgr2=0x7f85500a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:25.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.403+0000 7f856ba0f640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85500a2cf0 0x7f85500a51b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8564198ed0 tx=0x7f8554005a20 comp rx=0 tx=0).stop 2024-08-19T01:48:25.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.403+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8564100b50 msgr2=0x7f8564196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:25.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.403+0000 7f856ba0f640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8564100b50 0x7f8564196d70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f856000c920 tx=0x7f856000cde0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.408+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 shutdown_connections 2024-08-19T01:48:25.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.408+0000 7f856ba0f640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85500a2cf0 0x7f85500a51b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.408+0000 7f856ba0f640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8564102c20 0x7f85641977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.408+0000 7f856ba0f640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85641022a0 0x7f85641972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.408+0000 7f856ba0f640 1 --2- 172.21.15.64:0/3617536964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8564100b50 0x7f8564196d70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:25.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.408+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 >> 172.21.15.64:0/3617536964 conn(0x7f85640fc500 msgr2=0x7f8564106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:25.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.409+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 shutdown_connections 2024-08-19T01:48:25.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:25.411+0000 7f856ba0f640 1 -- 172.21.15.64:0/3617536964 wait complete. 2024-08-19T01:48:25.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 15.20 MB/sec warmup 117 sec latency 16132.529 ms 2024-08-19T01:48:26.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:25 smithi064 ceph-mon[18515]: pgmap v476: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 4.8 MiB/s wr, 170 op/s 2024-08-19T01:48:26.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:25 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3617536964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:26.153 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 3ee1542d --release --await' 2024-08-19T01:48:26.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 3ee1542d --release --await 2024-08-19T01:48:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:25 smithi123 ceph-mon[21829]: pgmap v476: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 4.8 MiB/s wr, 170 op/s 2024-08-19T01:48:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:25 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3617536964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:25 smithi161 ceph-mon[21757]: pgmap v476: 129 pgs: 129 active+clean; 968 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 4.8 MiB/s wr, 170 op/s 2024-08-19T01:48:26.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:25 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3617536964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:48:26.418 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:48:26.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 29761 15.07 MB/sec warmup 118 sec latency 17132.801 ms 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 -- 172.21.15.64:0/1756840839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 msgr2=0x7f5d40100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/1756840839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d40100c10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009930 tx=0x7f5d3002a420 comp rx=0 tx=0).stop 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 -- 172.21.15.64:0/1756840839 shutdown_connections 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/1756840839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d40101150 0x7f5d4010d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/1756840839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d40100c10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/1756840839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d401067b0 0x7f5d40106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 -- 172.21.15.64:0/1756840839 >> 172.21.15.64:0/1756840839 conn(0x7f5d400fc4d0 msgr2=0x7f5d400fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 -- 172.21.15.64:0/1756840839 shutdown_connections 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.883+0000 7f5d47ee3640 1 -- 172.21.15.64:0/1756840839 wait complete. 2024-08-19T01:48:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 Processor -- start 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 -- start start 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d4019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d40101150 0x7f5d4019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d401067b0 0x7f5d4019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d40110bc0 con 0x7f5d401007b0 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d40110a40 con 0x7f5d40101150 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d47ee3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d40110d40 con 0x7f5d401067b0 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d45c7f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d4019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d45c7f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d4019a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40772/0 (socket says 172.21.15.64:40772) 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d45c7f640 1 -- 172.21.15.64:0/3477877656 learned_addr learned my addr 172.21.15.64:0/3477877656 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.884+0000 7f5d4547e640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d40101150 0x7f5d4019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d46480640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d401067b0 0x7f5d4019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d45c7f640 1 -- 172.21.15.64:0/3477877656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d401067b0 msgr2=0x7f5d4019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d401067b0 0x7f5d4019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d45c7f640 1 -- 172.21.15.64:0/3477877656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d40101150 msgr2=0x7f5d4019a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d40101150 0x7f5d4019a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d45c7f640 1 -- 172.21.15.64:0/3477877656 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4019f3b0 con 0x7f5d401007b0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d46480640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d401067b0 0x7f5d4019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d4019a400 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d950 tx=0x7f5d3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d36ffd640 1 -- 172.21.15.64:0/3477877656 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c017070 con 0x7f5d401007b0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d36ffd640 1 -- 172.21.15.64:0/3477877656 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d3c009c30 con 0x7f5d401007b0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4019f6a0 con 0x7f5d401007b0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d36ffd640 1 -- 172.21.15.64:0/3477877656 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00f6b0 con 0x7f5d401007b0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.885+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d401a6f80 con 0x7f5d401007b0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.887+0000 7f5d36ffd640 1 -- 172.21.15.64:0/3477877656 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5d3c0109d0 con 0x7f5d401007b0 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 Processor -- start 2024-08-19T01:48:26.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 -- start start 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d36ffd640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5d2c0a2cf0 0x7f5d2c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf236d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf241630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf241b70 0x5599cf243f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5599cf19cc60 con 0x5599cf2372b0 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5599cf134b90 con 0x5599cf241b70 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5599cf198c20 con 0x5599cf236990 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d46480640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5d2c0a2cf0 0x7f5d2c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d36ffd640 1 -- 172.21.15.64:0/3477877656 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7f5d3c0cf400 con 0x7f5d401007b0 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4547e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf241630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4547e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf241630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40782/0 (socket says 172.21.15.64:40782) 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d4547e640 1 -- 172.21.15.64:0/3322922657 learned_addr learned my addr 172.21.15.64:0/3322922657 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf236d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.888+0000 7f5d46480640 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf241b70 0x5599cf243f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d4547e640 1 -- 172.21.15.64:0/3322922657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 msgr2=0x5599cf236d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d4547e640 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf236d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d4547e640 1 -- 172.21.15.64:0/3322922657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf241b70 msgr2=0x5599cf243f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d4547e640 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf241b70 0x5599cf243f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d4547e640 1 -- 172.21.15.64:0/3322922657 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ceeac580 con 0x5599cf2372b0 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d46480640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5d2c0a2cf0 0x7f5d2c0a51b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009270 tx=0x7f5d38013a80 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d4547e640 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf241630 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009930 tx=0x7f5d30005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a66ce923d6f17c5 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d2bfff640 1 -- 172.21.15.64:0/3322922657 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30035040 con 0x5599cf2372b0 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d2bfff640 1 -- 172.21.15.64:0/3322922657 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d30005520 con 0x5599cf2372b0 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.889+0000 7f5d2bfff640 1 -- 172.21.15.64:0/3322922657 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d300035f0 con 0x5599cf2372b0 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3322922657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 msgr2=0x5599cf241630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf241630 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009930 tx=0x7f5d30005c00 comp rx=0 tx=0).stop 2024-08-19T01:48:26.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3322922657 shutdown_connections 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf241b70 0x5599cf243f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf241630 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3322922657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf236d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3322922657 >> 172.21.15.64:0/3322922657 conn(0x5599cf232f20 msgr2=0x5599cf233300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3322922657 shutdown_connections 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3322922657 wait complete. 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 -- start start 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.890+0000 7f5d4a2e1740 1 Processor -- start 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf236990 0x5599cf23ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf23ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf23f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5599cf244740 con 0x5599cf236990 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5599cf13a7c0 con 0x5599cf2372b0 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5599cf2448c0 con 0x5599cf241b70 2024-08-19T01:48:26.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 10 client.0 _ll_register_callbacks cb 0x5599cf143f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf236990 0x5599cf23ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf236990 0x5599cf23ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40798/0 (socket says 172.21.15.64:40798) 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 -- 172.21.15.64:0/2873395304 learned_addr learned my addr 172.21.15.64:0/2873395304 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d46480640 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf23f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4547e640 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf23ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 -- 172.21.15.64:0/2873395304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 msgr2=0x5599cf23f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf23f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 -- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 msgr2=0x5599cf23ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf23ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 -- 172.21.15.64:0/2873395304 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599cef76960 con 0x5599cf236990 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf236990 0x5599cf23ea50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0ccd40 tx=0x7f5d3c09af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6a8602ed6105f7 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d297fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d297fa640 1 -- 172.21.15.64:0/2873395304 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c020070 con 0x5599cf236990 2024-08-19T01:48:26.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d297fa640 1 -- 172.21.15.64:0/2873395304 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7f5d3c097b80 con 0x5599cf236990 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.891+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d30005520 con 0x5599cf236990 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d297fa640 1 -- 172.21.15.64:0/2873395304 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5d3c067400 con 0x5599cf236990 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 10 client.16134 fetch_fsmap learned FSMap version 30 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5599cf250780 con 0x5599cf236990 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d297fa640 1 -- 172.21.15.64:0/2873395304 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d30005520 con 0x5599cf236990 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d297fa640 1 -- 172.21.15.64:0/2873395304 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5d3c0a3cf0 con 0x5599cf236990 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 10 client.16134 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 10 client.16134 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 20 client.16134 populate_metadata read hostname 'smithi064' 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf251be0 0x5599cf253ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 4 client.16134 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599cf1aa1d0 con 0x5599cf251be0 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.892+0000 7f5d4547e640 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf251be0 0x5599cf253ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.893+0000 7f5d4547e640 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf251be0 0x5599cf253ff0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d8fe3168cce84c1 server_cookie=23e5a438e36b09f3 in_seq=0 out_seq=0 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.893+0000 7f5d297fa640 10 client.16134 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.893+0000 7f5d297fa640 1 -- 172.21.15.64:0/2873395304 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5599cf1aa1d0 con 0x5599cf251be0 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.893+0000 7f5d297fa640 10 client.16134 handle_command_reply: tid=0 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.893+0000 7f5d4a2e1740 1 client.16134 shutdown 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.893+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf236990 msgr2=0x5599cf23ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.893+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf236990 0x5599cf23ea50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0ccd40 tx=0x7f5d3c09af40 comp rx=0 tx=0).stop 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 shutdown_connections 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf23f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf251be0 0x5599cf253ff0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf23ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/2873395304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf236990 0x5599cf23ea50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d297fa640 0 client.16134 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 >> 172.21.15.64:0/2873395304 conn(0x5599cf232f20 msgr2=0x5599cf238c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:26.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 shutdown_connections 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/2873395304 wait complete. 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.894+0000 7f5d4a2e1740 20 client.16134 trim_cache size 0 max 16384 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 Processor -- start 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 -- start start 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf236990 0x5599cf2393f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf2598c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf259e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d401b3960 con 0x5599cf2372b0 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d401b1230 con 0x5599cf236990 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d401af0a0 con 0x5599cf241b70 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf2598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf2598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40804/0 (socket says 172.21.15.64:40804) 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 -- 172.21.15.64:0/4067636689 learned_addr learned my addr 172.21.15.64:0/4067636689 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf236990 0x5599cf2393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 -- 172.21.15.64:0/4067636689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 msgr2=0x5599cf259e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d46480640 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf259e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf259e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 -- 172.21.15.64:0/4067636689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf236990 msgr2=0x5599cf2393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf236990 0x5599cf2393f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.911+0000 7f5d4547e640 1 -- 172.21.15.64:0/4067636689 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599cf233bb0 con 0x5599cf2372b0 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf236990 0x5599cf2393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d46480640 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf259e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4547e640 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf2598c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5d30004c70 tx=0x7f5d30004ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9689eab817361a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d2b7fe640 1 -- 172.21.15.64:0/4067636689 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30035030 con 0x5599cf2372b0 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d2b7fe640 1 -- 172.21.15.64:0/4067636689 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d30004980 con 0x5599cf2372b0 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d2b7fe640 1 -- 172.21.15.64:0/4067636689 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d300319d0 con 0x5599cf2372b0 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/4067636689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 msgr2=0x5599cf2598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf2598c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5d30004c70 tx=0x7f5d30004ca0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/4067636689 shutdown_connections 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf241b70 0x5599cf259e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf2372b0 0x5599cf2598c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/4067636689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf236990 0x5599cf2393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.912+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/4067636689 >> 172.21.15.64:0/4067636689 conn(0x5599cf232f20 msgr2=0x5599cf240ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/4067636689 shutdown_connections 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/4067636689 wait complete. 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 -- start start 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 Processor -- start 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf2458a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf245de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf241b70 0x5599cf246320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d401af6f0 con 0x5599cf241b70 2024-08-19T01:48:26.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d401b0b60 con 0x5599cf2372b0 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d401afdc0 con 0x5599cf236990 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d4a2e1740 10 client.0 _ll_register_callbacks cb 0x7f5d3c09ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d46480640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf241b70 0x5599cf246320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d46480640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf241b70 0x5599cf246320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40818/0 (socket says 172.21.15.64:40818) 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.913+0000 7f5d46480640 1 -- 172.21.15.64:0/3850082159 learned_addr learned my addr 172.21.15.64:0/3850082159 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf2458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d4547e640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf245de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d46480640 1 -- 172.21.15.64:0/3850082159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 msgr2=0x5599cf2458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d46480640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf2458a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d46480640 1 -- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 msgr2=0x5599cf245de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d46480640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf245de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d46480640 1 -- 172.21.15.64:0/3850082159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599cf2bb890 con 0x5599cf241b70 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d4547e640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf245de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:48:26.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d46480640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf241b70 0x5599cf246320 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5d380118f0 tx=0x7f5d38019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1637ef350bc6219 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d297fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d297fa640 1 -- 172.21.15.64:0/3850082159 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38013220 con 0x5599cf241b70 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d297fa640 1 -- 172.21.15.64:0/3850082159 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7f5d38004b20 con 0x5599cf241b70 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d30004980 con 0x5599cf241b70 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d297fa640 1 -- 172.21.15.64:0/3850082159 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5d38033050 con 0x5599cf241b70 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d4a2e1740 10 client.16140 fetch_fsmap learned FSMap version 30 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.914+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5599cf2bcb00 con 0x5599cf241b70 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d297fa640 1 -- 172.21.15.64:0/3850082159 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d30004980 con 0x5599cf241b70 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d297fa640 1 -- 172.21.15.64:0/3850082159 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5d380126f0 con 0x5599cf241b70 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d4a2e1740 10 client.16140 fetch_fsmap finished waiting for FSMap version 30 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d4a2e1740 10 client.16140 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d4a2e1740 20 client.16140 populate_metadata read hostname 'smithi064' 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf2bd330 0x5599cf2bf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d4a2e1740 4 client.16140 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3ee1542d", "release": true, "await": true}] 2024-08-19T01:48:26.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3ee1542d", "release": true, "await": true}) -- 0x5599cf1a46e0 con 0x5599cf2bd330 2024-08-19T01:48:26.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf2bd330 0x5599cf2bf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:48:26.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d297fa640 10 client.16140 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:26.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.915+0000 7f5d45c7f640 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf2bd330 0x5599cf2bf750 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d237c4294daa119 server_cookie=ab23f04bdc5d16b5 in_seq=0 out_seq=0 2024-08-19T01:48:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d297fa640 1 -- 172.21.15.64:0/3850082159 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5599cf1a46e0 con 0x5599cf2bd330 2024-08-19T01:48:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d297fa640 10 client.16140 handle_command_reply: tid=0 2024-08-19T01:48:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 client.16140 shutdown 2024-08-19T01:48:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf241b70 msgr2=0x5599cf246320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf241b70 0x5599cf246320 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5d380118f0 tx=0x7f5d38019000 comp rx=0 tx=0).stop 2024-08-19T01:48:26.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 shutdown_connections 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599cf241b70 0x5599cf246320 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x5599cf2bd330 0x5599cf2bf750 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5599cf2372b0 0x5599cf245de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 --2- 172.21.15.64:0/3850082159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5599cf236990 0x5599cf2458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d297fa640 0 client.16140 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 >> 172.21.15.64:0/3850082159 conn(0x5599cf232f20 msgr2=0x5599cf240ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 shutdown_connections 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 1 -- 172.21.15.64:0/3850082159 wait complete. 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.989+0000 7f5d4a2e1740 20 client.16140 trim_cache size 0 max 16384 2024-08-19T01:48:26.991 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 56, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "3ee1542d": { 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 56, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 137.7, 2024-08-19T01:48:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:48:26.993 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:48:26.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5d2c0a2cf0 msgr2=0x7f5d2c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5d2c0a2cf0 0x7f5d2c0a51b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009270 tx=0x7f5d38013a80 comp rx=0 tx=0).stop 2024-08-19T01:48:26.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 msgr2=0x7f5d4019a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:48:26.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d4019a400 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d950 tx=0x7f5d3c00de10 comp rx=0 tx=0).stop 2024-08-19T01:48:26.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 shutdown_connections 2024-08-19T01:48:26.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f5d2c0a2cf0 0x7f5d2c0a51b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d401067b0 0x7f5d4019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d40101150 0x7f5d4019a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 --2- 172.21.15.64:0/3477877656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d401007b0 0x7f5d4019a400 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:48:26.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 >> 172.21.15.64:0/3477877656 conn(0x7f5d400fc4d0 msgr2=0x7f5d4006a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:48:26.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 shutdown_connections 2024-08-19T01:48:26.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:48:26.990+0000 7f5d47ee3640 1 -- 172.21.15.64:0/3477877656 wait complete. 2024-08-19T01:48:27.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 30130 15.12 MB/sec warmup 119 sec latency 17435.976 ms 2024-08-19T01:48:27.858 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3ee1542d', total seconds elapsed: 21.1 2024-08-19T01:48:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:27 smithi123 ceph-mon[21829]: pgmap v477: 129 pgs: 129 active+clean; 967 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 4.7 MiB/s wr, 176 op/s 2024-08-19T01:48:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:27 smithi161 ceph-mon[21757]: pgmap v477: 129 pgs: 129 active+clean; 967 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 4.7 MiB/s wr, 176 op/s 2024-08-19T01:48:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:27 smithi064 ceph-mon[18515]: pgmap v477: 129 pgs: 129 active+clean; 967 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 4.7 MiB/s wr, 176 op/s 2024-08-19T01:48:29.723 INFO:tasks.workunit.client.0.smithi064.stdout: 10 31203 23.82 MB/sec execute 1 sec latency 69.209 ms 2024-08-19T01:48:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:29 smithi123 ceph-mon[21829]: pgmap v478: 129 pgs: 129 active+clean; 967 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 2.8 MiB/s wr, 116 op/s 2024-08-19T01:48:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:29 smithi161 ceph-mon[21757]: pgmap v478: 129 pgs: 129 active+clean; 967 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 2.8 MiB/s wr, 116 op/s 2024-08-19T01:48:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:29 smithi064 ceph-mon[18515]: pgmap v478: 129 pgs: 129 active+clean; 967 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 2.8 MiB/s wr, 116 op/s 2024-08-19T01:48:30.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 31805 33.10 MB/sec execute 2 sec latency 53.617 ms 2024-08-19T01:48:31.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 32290 29.53 MB/sec execute 3 sec latency 88.931 ms 2024-08-19T01:48:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:31 smithi123 ceph-mon[21829]: pgmap v479: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 233 op/s 2024-08-19T01:48:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:31 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:48:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:31 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:48:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:31 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:48:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:31 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:48:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:31 smithi161 ceph-mon[21757]: pgmap v479: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 233 op/s 2024-08-19T01:48:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:31 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:48:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:31 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:48:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:31 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:48:32.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:31 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:48:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:31 smithi064 ceph-mon[18515]: pgmap v479: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 233 op/s 2024-08-19T01:48:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:31 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:48:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:31 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:48:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:31 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:48:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:31 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:48:32.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 32853 29.32 MB/sec execute 4 sec latency 77.263 ms 2024-08-19T01:48:33.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 33563 32.06 MB/sec execute 5 sec latency 74.836 ms 2024-08-19T01:48:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:33 smithi123 ceph-mon[21829]: pgmap v480: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-08-19T01:48:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:33 smithi161 ceph-mon[21757]: pgmap v480: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-08-19T01:48:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:33 smithi064 ceph-mon[18515]: pgmap v480: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-08-19T01:48:34.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 34178 32.96 MB/sec execute 6 sec latency 59.925 ms 2024-08-19T01:48:35.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 34564 31.21 MB/sec execute 7 sec latency 138.588 ms 2024-08-19T01:48:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:35 smithi123 ceph-mon[21829]: pgmap v481: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-08-19T01:48:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:35 smithi161 ceph-mon[21757]: pgmap v481: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-08-19T01:48:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:35 smithi064 ceph-mon[18515]: pgmap v481: 129 pgs: 129 active+clean; 982 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 147 op/s 2024-08-19T01:48:36.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 35146 30.39 MB/sec execute 8 sec latency 89.171 ms 2024-08-19T01:48:37.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 35746 30.62 MB/sec execute 9 sec latency 51.332 ms 2024-08-19T01:48:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:37 smithi123 ceph-mon[21829]: pgmap v482: 129 pgs: 129 active+clean; 1004 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.2 MiB/s wr, 311 op/s 2024-08-19T01:48:38.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:37 smithi161 ceph-mon[21757]: pgmap v482: 129 pgs: 129 active+clean; 1004 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.2 MiB/s wr, 311 op/s 2024-08-19T01:48:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:37 smithi064 ceph-mon[18515]: pgmap v482: 129 pgs: 129 active+clean; 1004 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.2 MiB/s wr, 311 op/s 2024-08-19T01:48:38.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 36264 29.76 MB/sec execute 10 sec latency 104.734 ms 2024-08-19T01:48:39.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 36912 30.46 MB/sec execute 11 sec latency 105.279 ms 2024-08-19T01:48:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:39 smithi123 ceph-mon[21829]: pgmap v483: 129 pgs: 129 active+clean; 1004 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-08-19T01:48:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:39 smithi161 ceph-mon[21757]: pgmap v483: 129 pgs: 129 active+clean; 1004 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-08-19T01:48:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:39 smithi064 ceph-mon[18515]: pgmap v483: 129 pgs: 129 active+clean; 1004 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-08-19T01:48:40.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 37548 31.34 MB/sec execute 12 sec latency 57.088 ms 2024-08-19T01:48:41.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 37972 31.33 MB/sec execute 13 sec latency 117.385 ms 2024-08-19T01:48:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:41 smithi123 ceph-mon[21829]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 466 op/s 2024-08-19T01:48:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:41 smithi161 ceph-mon[21757]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 466 op/s 2024-08-19T01:48:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:41 smithi064 ceph-mon[18515]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 466 op/s 2024-08-19T01:48:42.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 38491 30.66 MB/sec execute 14 sec latency 75.606 ms 2024-08-19T01:48:43.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 38964 29.75 MB/sec execute 15 sec latency 87.182 ms 2024-08-19T01:48:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:43 smithi123 ceph-mon[21829]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.3 MiB/s wr, 350 op/s 2024-08-19T01:48:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:43 smithi161 ceph-mon[21757]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.3 MiB/s wr, 350 op/s 2024-08-19T01:48:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:43 smithi064 ceph-mon[18515]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.3 MiB/s wr, 350 op/s 2024-08-19T01:48:44.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 39594 30.54 MB/sec execute 16 sec latency 69.807 ms 2024-08-19T01:48:45.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 40119 30.22 MB/sec execute 17 sec latency 52.226 ms 2024-08-19T01:48:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:45 smithi123 ceph-mon[21829]: pgmap v486: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.3 MiB/s wr, 350 op/s 2024-08-19T01:48:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:45 smithi161 ceph-mon[21757]: pgmap v486: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.3 MiB/s wr, 350 op/s 2024-08-19T01:48:46.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:45 smithi064 ceph-mon[18515]: pgmap v486: 129 pgs: 129 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.3 MiB/s wr, 350 op/s 2024-08-19T01:48:46.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 40739 30.12 MB/sec execute 18 sec latency 74.081 ms 2024-08-19T01:48:47.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:46 smithi123 ceph-mon[21829]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.1 MiB/s wr, 539 op/s 2024-08-19T01:48:47.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:46 smithi161 ceph-mon[21757]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.1 MiB/s wr, 539 op/s 2024-08-19T01:48:47.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:46 smithi064 ceph-mon[18515]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.1 MiB/s wr, 539 op/s 2024-08-19T01:48:47.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 41451 30.50 MB/sec execute 19 sec latency 68.641 ms 2024-08-19T01:48:48.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 42089 30.99 MB/sec execute 20 sec latency 214.423 ms 2024-08-19T01:48:49.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:49 smithi123 ceph-mon[21829]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 375 op/s 2024-08-19T01:48:49.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:49 smithi161 ceph-mon[21757]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 375 op/s 2024-08-19T01:48:49.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:49 smithi064 ceph-mon[18515]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 375 op/s 2024-08-19T01:48:49.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 42616 31.16 MB/sec execute 21 sec latency 71.302 ms 2024-08-19T01:48:50.724 INFO:tasks.workunit.client.0.smithi064.stdout: 10 43127 30.58 MB/sec execute 22 sec latency 104.209 ms 2024-08-19T01:48:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:51 smithi123 ceph-mon[21829]: pgmap v489: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 536 op/s 2024-08-19T01:48:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:51 smithi161 ceph-mon[21757]: pgmap v489: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 536 op/s 2024-08-19T01:48:51.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:51 smithi064 ceph-mon[18515]: pgmap v489: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 536 op/s 2024-08-19T01:48:51.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 43659 30.56 MB/sec execute 23 sec latency 65.659 ms 2024-08-19T01:48:52.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 44352 30.94 MB/sec execute 24 sec latency 54.747 ms 2024-08-19T01:48:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:53 smithi123 ceph-mon[21829]: pgmap v490: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.0 MiB/s wr, 349 op/s 2024-08-19T01:48:53.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:53 smithi161 ceph-mon[21757]: pgmap v490: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.0 MiB/s wr, 349 op/s 2024-08-19T01:48:53.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:53 smithi064 ceph-mon[18515]: pgmap v490: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.0 MiB/s wr, 349 op/s 2024-08-19T01:48:53.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 44990 30.86 MB/sec execute 25 sec latency 64.279 ms 2024-08-19T01:48:54.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 45596 30.99 MB/sec execute 26 sec latency 55.136 ms 2024-08-19T01:48:55.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:55 smithi123 ceph-mon[21829]: pgmap v491: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.0 MiB/s wr, 349 op/s 2024-08-19T01:48:55.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:55 smithi161 ceph-mon[21757]: pgmap v491: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.0 MiB/s wr, 349 op/s 2024-08-19T01:48:55.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:55 smithi064 ceph-mon[18515]: pgmap v491: 129 pgs: 129 active+clean; 1.0 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.0 MiB/s wr, 349 op/s 2024-08-19T01:48:55.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 46171 31.10 MB/sec execute 27 sec latency 68.860 ms 2024-08-19T01:48:56.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 46817 31.24 MB/sec execute 28 sec latency 65.577 ms 2024-08-19T01:48:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:57 smithi123 ceph-mon[21829]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 504 op/s 2024-08-19T01:48:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:57 smithi161 ceph-mon[21757]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 504 op/s 2024-08-19T01:48:57.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:57 smithi064 ceph-mon[18515]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 504 op/s 2024-08-19T01:48:57.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 47441 31.30 MB/sec execute 29 sec latency 90.466 ms 2024-08-19T01:48:58.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:58 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:48:58.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:58 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:48:58.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:58 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:48:58.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 47872 31.19 MB/sec execute 30 sec latency 114.506 ms 2024-08-19T01:48:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:48:59 smithi123 ceph-mon[21829]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.5 MiB/s wr, 315 op/s 2024-08-19T01:48:59.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:48:59 smithi161 ceph-mon[21757]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.5 MiB/s wr, 315 op/s 2024-08-19T01:48:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:48:59 smithi064 ceph-mon[18515]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.5 MiB/s wr, 315 op/s 2024-08-19T01:48:59.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48307 30.92 MB/sec execute 31 sec latency 146.942 ms 2024-08-19T01:49:00.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48577 30.33 MB/sec execute 32 sec latency 575.459 ms 2024-08-19T01:49:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:01 smithi123 ceph-mon[21829]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.6 MiB/s wr, 499 op/s 2024-08-19T01:49:01.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:01 smithi161 ceph-mon[21757]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.6 MiB/s wr, 499 op/s 2024-08-19T01:49:01.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:01 smithi064 ceph-mon[18515]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.6 MiB/s wr, 499 op/s 2024-08-19T01:49:01.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48577 29.41 MB/sec execute 33 sec latency 1575.562 ms 2024-08-19T01:49:02.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48577 28.55 MB/sec execute 34 sec latency 2575.661 ms 2024-08-19T01:49:03.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:03 smithi123 ceph-mon[21829]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:49:03.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:03 smithi161 ceph-mon[21757]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:49:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:03 smithi064 ceph-mon[18515]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:49:03.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48577 27.73 MB/sec execute 35 sec latency 3575.776 ms 2024-08-19T01:49:04.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48577 26.96 MB/sec execute 36 sec latency 4575.914 ms 2024-08-19T01:49:05.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:05 smithi123 ceph-mon[21829]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:49:05.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:05 smithi161 ceph-mon[21757]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:49:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:05 smithi064 ceph-mon[18515]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.4 MiB/s wr, 338 op/s 2024-08-19T01:49:05.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48675 26.29 MB/sec execute 37 sec latency 4881.406 ms 2024-08-19T01:49:06.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 48758 25.62 MB/sec execute 38 sec latency 1473.573 ms 2024-08-19T01:49:07.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:07 smithi123 ceph-mon[21829]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 7.8 MiB/s wr, 408 op/s 2024-08-19T01:49:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:07 smithi161 ceph-mon[21757]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 7.8 MiB/s wr, 408 op/s 2024-08-19T01:49:07.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:07 smithi064 ceph-mon[18515]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 7.8 MiB/s wr, 408 op/s 2024-08-19T01:49:07.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 49268 25.59 MB/sec execute 39 sec latency 160.463 ms 2024-08-19T01:49:08.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:08 smithi123 ceph-mon[21829]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:49:08.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:08 smithi161 ceph-mon[21757]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:49:08.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:08 smithi064 ceph-mon[18515]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-08-19T01:49:08.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 49648 25.39 MB/sec execute 40 sec latency 191.144 ms 2024-08-19T01:49:09.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:09 smithi123 ceph-mon[21829]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.6 MiB/s wr, 253 op/s 2024-08-19T01:49:09.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:09 smithi161 ceph-mon[21757]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.6 MiB/s wr, 253 op/s 2024-08-19T01:49:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:09 smithi064 ceph-mon[18515]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.6 MiB/s wr, 253 op/s 2024-08-19T01:49:09.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 49923 25.29 MB/sec execute 41 sec latency 401.497 ms 2024-08-19T01:49:10.725 INFO:tasks.workunit.client.0.smithi064.stdout: 10 50359 25.55 MB/sec execute 42 sec latency 358.546 ms 2024-08-19T01:49:11.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:11 smithi123 ceph-mon[21829]: pgmap v499: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.9 MiB/s wr, 555 op/s 2024-08-19T01:49:11.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:11 smithi161 ceph-mon[21757]: pgmap v499: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.9 MiB/s wr, 555 op/s 2024-08-19T01:49:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:11 smithi064 ceph-mon[18515]: pgmap v499: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.9 MiB/s wr, 555 op/s 2024-08-19T01:49:11.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 50778 25.42 MB/sec execute 43 sec latency 92.064 ms 2024-08-19T01:49:12.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 51388 25.38 MB/sec execute 44 sec latency 101.412 ms 2024-08-19T01:49:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:13 smithi161 ceph-mon[21757]: pgmap v500: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 371 op/s 2024-08-19T01:49:13.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:13 smithi161 ceph-mon[21757]: osdmap e113: 12 total, 12 up, 12 in 2024-08-19T01:49:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:13 smithi064 ceph-mon[18515]: pgmap v500: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 371 op/s 2024-08-19T01:49:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:13 smithi064 ceph-mon[18515]: osdmap e113: 12 total, 12 up, 12 in 2024-08-19T01:49:13.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:13 smithi123 ceph-mon[21829]: pgmap v500: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 371 op/s 2024-08-19T01:49:13.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:13 smithi123 ceph-mon[21829]: osdmap e113: 12 total, 12 up, 12 in 2024-08-19T01:49:13.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 51960 25.58 MB/sec execute 45 sec latency 64.677 ms 2024-08-19T01:49:14.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 52349 25.47 MB/sec execute 46 sec latency 323.875 ms 2024-08-19T01:49:15.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:15 smithi161 ceph-mon[21757]: pgmap v502: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 445 op/s 2024-08-19T01:49:15.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:15 smithi064 ceph-mon[18515]: pgmap v502: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 445 op/s 2024-08-19T01:49:15.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:15 smithi123 ceph-mon[21829]: pgmap v502: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 445 op/s 2024-08-19T01:49:15.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 52891 25.55 MB/sec execute 47 sec latency 76.382 ms 2024-08-19T01:49:16.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 53499 25.76 MB/sec execute 48 sec latency 114.659 ms 2024-08-19T01:49:17.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:17 smithi161 ceph-mon[21757]: pgmap v503: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-08-19T01:49:17.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:17 smithi064 ceph-mon[18515]: pgmap v503: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-08-19T01:49:17.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:17 smithi123 ceph-mon[21829]: pgmap v503: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-08-19T01:49:17.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 54066 25.76 MB/sec execute 49 sec latency 105.053 ms 2024-08-19T01:49:18.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 54637 25.86 MB/sec execute 50 sec latency 83.824 ms 2024-08-19T01:49:19.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 55076 25.89 MB/sec execute 51 sec latency 268.094 ms 2024-08-19T01:49:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:19 smithi123 ceph-mon[21829]: pgmap v504: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-08-19T01:49:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:19 smithi123 ceph-mon[21829]: osdmap e114: 12 total, 12 up, 12 in 2024-08-19T01:49:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:19 smithi161 ceph-mon[21757]: pgmap v504: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-08-19T01:49:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:19 smithi161 ceph-mon[21757]: osdmap e114: 12 total, 12 up, 12 in 2024-08-19T01:49:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:19 smithi064 ceph-mon[18515]: pgmap v504: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-08-19T01:49:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:19 smithi064 ceph-mon[18515]: osdmap e114: 12 total, 12 up, 12 in 2024-08-19T01:49:20.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 55603 26.02 MB/sec execute 52 sec latency 135.025 ms 2024-08-19T01:49:21.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 56022 25.94 MB/sec execute 53 sec latency 134.663 ms 2024-08-19T01:49:22.123 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:21 smithi064 ceph-mon[18515]: pgmap v506: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 676 op/s 2024-08-19T01:49:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:21 smithi123 ceph-mon[21829]: pgmap v506: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 676 op/s 2024-08-19T01:49:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:21 smithi161 ceph-mon[21757]: pgmap v506: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 676 op/s 2024-08-19T01:49:22.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 56459 25.74 MB/sec execute 54 sec latency 168.015 ms 2024-08-19T01:49:23.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 57021 25.84 MB/sec execute 55 sec latency 88.594 ms 2024-08-19T01:49:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:23 smithi123 ceph-mon[21829]: pgmap v507: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 11 MiB/s wr, 546 op/s 2024-08-19T01:49:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:23 smithi161 ceph-mon[21757]: pgmap v507: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 11 MiB/s wr, 546 op/s 2024-08-19T01:49:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:23 smithi064 ceph-mon[18515]: pgmap v507: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 11 MiB/s wr, 546 op/s 2024-08-19T01:49:24.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 57485 25.83 MB/sec execute 56 sec latency 249.413 ms 2024-08-19T01:49:25.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 58033 25.87 MB/sec execute 57 sec latency 108.258 ms 2024-08-19T01:49:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:25 smithi123 ceph-mon[21829]: pgmap v508: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 11 MiB/s wr, 541 op/s 2024-08-19T01:49:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:25 smithi161 ceph-mon[21757]: pgmap v508: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 11 MiB/s wr, 541 op/s 2024-08-19T01:49:26.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:25 smithi064 ceph-mon[18515]: pgmap v508: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 11 MiB/s wr, 541 op/s 2024-08-19T01:49:26.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 58468 25.96 MB/sec execute 58 sec latency 367.768 ms 2024-08-19T01:49:27.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 59052 26.06 MB/sec execute 59 sec latency 61.302 ms 2024-08-19T01:49:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:27 smithi123 ceph-mon[21829]: pgmap v509: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 542 op/s 2024-08-19T01:49:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:27 smithi161 ceph-mon[21757]: pgmap v509: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 542 op/s 2024-08-19T01:49:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:27 smithi064 ceph-mon[18515]: pgmap v509: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 542 op/s 2024-08-19T01:49:28.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 59536 26.04 MB/sec execute 60 sec latency 81.061 ms 2024-08-19T01:49:29.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 59906 25.90 MB/sec execute 61 sec latency 236.054 ms 2024-08-19T01:49:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:29 smithi123 ceph-mon[21829]: pgmap v510: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 542 op/s 2024-08-19T01:49:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:29 smithi161 ceph-mon[21757]: pgmap v510: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 542 op/s 2024-08-19T01:49:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:29 smithi064 ceph-mon[18515]: pgmap v510: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 542 op/s 2024-08-19T01:49:30.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 60480 25.95 MB/sec execute 62 sec latency 130.632 ms 2024-08-19T01:49:31.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 61071 26.00 MB/sec execute 63 sec latency 84.562 ms 2024-08-19T01:49:32.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:31 smithi123 ceph-mon[21829]: pgmap v511: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 694 op/s 2024-08-19T01:49:32.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:31 smithi161 ceph-mon[21757]: pgmap v511: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 694 op/s 2024-08-19T01:49:32.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:31 smithi064 ceph-mon[18515]: pgmap v511: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 694 op/s 2024-08-19T01:49:32.726 INFO:tasks.workunit.client.0.smithi064.stdout: 10 61618 26.17 MB/sec execute 64 sec latency 214.608 ms 2024-08-19T01:49:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:32 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:49:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:32 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:49:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:32 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:49:33.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:32 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:49:33.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:32 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:49:33.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:32 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:49:33.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:32 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:49:33.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:32 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:49:33.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:32 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:49:33.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:32 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:49:33.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:32 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:49:33.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:32 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:49:33.416 INFO:tasks.quiescer.fs.[cephfs]:Total duration (54.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-08-19T01:49:33.416 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.05 2024-08-19T01:49:33.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:49:33.706 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:49:33.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 62134 26.13 MB/sec execute 65 sec latency 149.575 ms 2024-08-19T01:49:34.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:33 smithi123 ceph-mon[21829]: pgmap v512: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.4 MiB/s wr, 423 op/s 2024-08-19T01:49:34.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:33 smithi161 ceph-mon[21757]: pgmap v512: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.4 MiB/s wr, 423 op/s 2024-08-19T01:49:34.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:33 smithi064 ceph-mon[18515]: pgmap v512: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.4 MiB/s wr, 423 op/s 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 -- 172.21.15.64:0/171101008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1078e0 msgr2=0x7faacc107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 --2- 172.21.15.64:0/171101008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1078e0 0x7faacc107cc0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7faabc009930 tx=0x7faabc02a420 comp rx=0 tx=0).stop 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 -- 172.21.15.64:0/171101008 shutdown_connections 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 --2- 172.21.15.64:0/171101008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faacc102280 0x7faacc10e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 --2- 172.21.15.64:0/171101008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faacc1018e0 0x7faacc101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 --2- 172.21.15.64:0/171101008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1078e0 0x7faacc107cc0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 -- 172.21.15.64:0/171101008 >> 172.21.15.64:0/171101008 conn(0x7faacc0fd600 msgr2=0x7faacc0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 -- 172.21.15.64:0/171101008 shutdown_connections 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.447+0000 7faad2252640 1 -- 172.21.15.64:0/171101008 wait complete. 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.448+0000 7faad2252640 1 Processor -- start 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.448+0000 7faad2252640 1 -- start start 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.448+0000 7faad2252640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1018e0 0x7faacc074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.448+0000 7faad2252640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faacc102280 0x7faacc074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.448+0000 7faad2252640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faacc1078e0 0x7faacc074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faad2252640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faacc111ca0 con 0x7faacc1018e0 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faad2252640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faacc111b20 con 0x7faacc102280 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faad2252640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faacc111e20 con 0x7faacc1078e0 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1018e0 0x7faacc074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faacc102280 0x7faacc074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1018e0 0x7faacc074220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45654/0 (socket says 172.21.15.64:45654) 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 -- 172.21.15.64:0/23448357 learned_addr learned my addr 172.21.15.64:0/23448357 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacbfff640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faacc1078e0 0x7faacc074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 -- 172.21.15.64:0/23448357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faacc1078e0 msgr2=0x7faacc074ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faacc1078e0 0x7faacc074ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 -- 172.21.15.64:0/23448357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faacc102280 msgr2=0x7faacc074760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faacc102280 0x7faacc074760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 -- 172.21.15.64:0/23448357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faacc1a66a0 con 0x7faacc1018e0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacaffd640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faacc102280 0x7faacc074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faacb7fe640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1018e0 0x7faacc074220 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7faabc009230 tx=0x7faabc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faac8ff9640 1 -- 172.21.15.64:0/23448357 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc03f030 con 0x7faacc1018e0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.449+0000 7faac8ff9640 1 -- 172.21.15.64:0/23448357 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faabc003ad0 con 0x7faacc1018e0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.450+0000 7faac8ff9640 1 -- 172.21.15.64:0/23448357 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc035070 con 0x7faacc1018e0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.450+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faacc1a6930 con 0x7faacc1018e0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.450+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faacc1a6e10 con 0x7faacc1018e0 2024-08-19T01:49:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.451+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faacc101d40 con 0x7faacc1018e0 2024-08-19T01:49:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.452+0000 7faac8ff9640 1 -- 172.21.15.64:0/23448357 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7faabc042030 con 0x7faacc1018e0 2024-08-19T01:49:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.453+0000 7faac8ff9640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faab40a2cf0 0x7faab40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.453+0000 7faac8ff9640 1 -- 172.21.15.64:0/23448357 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7faabc0f0230 con 0x7faacc1018e0 2024-08-19T01:49:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.454+0000 7faacaffd640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faab40a2cf0 0x7faab40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.454+0000 7faacaffd640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faab40a2cf0 0x7faab40a51b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faacc075cb0 tx=0x7faac000b410 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.460+0000 7faac8ff9640 1 -- 172.21.15.64:0/23448357 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7faabc0bc3f0 con 0x7faacc1018e0 2024-08-19T01:49:34.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 62663 26.21 MB/sec execute 66 sec latency 156.013 ms 2024-08-19T01:49:34.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.787+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faacc075750 con 0x7faacc1018e0 2024-08-19T01:49:34.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.790+0000 7faac8ff9640 1 -- 172.21.15.64:0/23448357 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11921 (secure 0 0 0) 0x7faabc0c06f0 con 0x7faacc1018e0 2024-08-19T01:49:34.793 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:49:34.793 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":32,"btime":"2024-08-19T01:49:07:146032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:49:06.572445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:49:34.793 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 32 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.793+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faab40a2cf0 msgr2=0x7faab40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.793+0000 7faad2252640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faab40a2cf0 0x7faab40a51b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faacc075cb0 tx=0x7faac000b410 comp rx=0 tx=0).stop 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1018e0 msgr2=0x7faacc074220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1018e0 0x7faacc074220 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7faabc009230 tx=0x7faabc004270 comp rx=0 tx=0).stop 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 shutdown_connections 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7faab40a2cf0 0x7faab40a51b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faacc1078e0 0x7faacc074ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faacc102280 0x7faacc074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 --2- 172.21.15.64:0/23448357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faacc1018e0 0x7faacc074220 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:34.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.794+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 >> 172.21.15.64:0/23448357 conn(0x7faacc0fd600 msgr2=0x7faacc0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.795+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 shutdown_connections 2024-08-19T01:49:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:34.795+0000 7faad2252640 1 -- 172.21.15.64:0/23448357 wait complete. 2024-08-19T01:49:35.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:34 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/23448357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:49:35.178 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:34 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/23448357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:49:35.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:34 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/23448357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:49:35.669 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0' 2024-08-19T01:49:35.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0 2024-08-19T01:49:35.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63276 26.33 MB/sec execute 67 sec latency 54.647 ms 2024-08-19T01:49:35.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:35 smithi064 ceph-mon[18515]: pgmap v513: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.4 MiB/s wr, 423 op/s 2024-08-19T01:49:35.990 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:49:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:35 smithi123 ceph-mon[21829]: pgmap v513: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.4 MiB/s wr, 423 op/s 2024-08-19T01:49:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:35 smithi161 ceph-mon[21757]: pgmap v513: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.4 MiB/s wr, 423 op/s 2024-08-19T01:49:36.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.630+0000 7f8d68229640 1 -- 172.21.15.64:0/3414316701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d601007b0 msgr2=0x7f8d60100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:36.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.630+0000 7f8d68229640 1 --2- 172.21.15.64:0/3414316701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d601007b0 0x7f8d60100c10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009930 tx=0x7f8d5002a420 comp rx=0 tx=0).stop 2024-08-19T01:49:36.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.630+0000 7f8d68229640 1 -- 172.21.15.64:0/3414316701 shutdown_connections 2024-08-19T01:49:36.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.630+0000 7f8d68229640 1 --2- 172.21.15.64:0/3414316701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d60101150 0x7f8d6010d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.630+0000 7f8d68229640 1 --2- 172.21.15.64:0/3414316701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d601007b0 0x7f8d60100c10 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.630+0000 7f8d68229640 1 --2- 172.21.15.64:0/3414316701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d601067b0 0x7f8d60106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.630+0000 7f8d68229640 1 -- 172.21.15.64:0/3414316701 >> 172.21.15.64:0/3414316701 conn(0x7f8d600fc4d0 msgr2=0x7f8d600fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.631+0000 7f8d68229640 1 -- 172.21.15.64:0/3414316701 shutdown_connections 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.631+0000 7f8d68229640 1 -- 172.21.15.64:0/3414316701 wait complete. 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.631+0000 7f8d68229640 1 Processor -- start 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.631+0000 7f8d68229640 1 -- start start 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d68229640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d601007b0 0x7f8d60198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d68229640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d60101150 0x7f8d60198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d68229640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d601067b0 0x7f8d6019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d68229640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8d60110c00 con 0x7f8d60101150 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d68229640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d60110a80 con 0x7f8d601007b0 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d68229640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8d60110d80 con 0x7f8d601067b0 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d667c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d601067b0 0x7f8d6019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d657c4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d60101150 0x7f8d60198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d667c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d601067b0 0x7f8d6019ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:55676/0 (socket says 172.21.15.64:55676) 2024-08-19T01:49:36.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d667c6640 1 -- 172.21.15.64:0/1290440316 learned_addr learned my addr 172.21.15.64:0/1290440316 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:49:36.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d657c4640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d60101150 0x7f8d60198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45674/0 (socket says 172.21.15.64:45674) 2024-08-19T01:49:36.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d601007b0 0x7f8d60198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.632+0000 7f8d667c6640 1 -- 172.21.15.64:0/1290440316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d601007b0 msgr2=0x7f8d60198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d667c6640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d601007b0 0x7f8d60198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d667c6640 1 -- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d60101150 msgr2=0x7f8d60198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d667c6640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d60101150 0x7f8d60198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d667c6640 1 -- 172.21.15.64:0/1290440316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d6019d100 con 0x7f8d601067b0 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d601007b0 0x7f8d60198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d657c4640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d60101150 0x7f8d60198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d667c6640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d601067b0 0x7f8d6019ca20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800eb10 tx=0x7f8d5800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:36.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d56ffd640 1 -- 172.21.15.64:0/1290440316 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58004b20 con 0x7f8d601067b0 2024-08-19T01:49:36.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d56ffd640 1 -- 172.21.15.64:0/1290440316 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d58004cc0 con 0x7f8d601067b0 2024-08-19T01:49:36.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d6019d3f0 con 0x7f8d601067b0 2024-08-19T01:49:36.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d601a4cd0 con 0x7f8d601067b0 2024-08-19T01:49:36.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.633+0000 7f8d56ffd640 1 -- 172.21.15.64:0/1290440316 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58010760 con 0x7f8d601067b0 2024-08-19T01:49:36.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.636+0000 7f8d56ffd640 1 -- 172.21.15.64:0/1290440316 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8d58005000 con 0x7f8d601067b0 2024-08-19T01:49:36.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.636+0000 7f8d56ffd640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8d4c0a2cf0 0x7f8d4c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 Processor -- start 2024-08-19T01:49:36.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d56ffd640 1 -- 172.21.15.64:0/1290440316 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7f8d580d00c0 con 0x7f8d601067b0 2024-08-19T01:49:36.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8d4c0a2cf0 0x7f8d4c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 -- start start 2024-08-19T01:49:36.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d840580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d851ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55780d7a5810 con 0x55780d84f8b0 2024-08-19T01:49:36.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55780d74cc70 con 0x55780d840ac0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d6a627740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55780d7a17d0 con 0x55780d8401a0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d657c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8d4c0a2cf0 0x7f8d4c0a51b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009ba0 tx=0x7f8d5c005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d657c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:57762/0 (socket says 172.21.15.64:57762) 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d667c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d851ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d657c4640 1 -- 172.21.15.64:0/374071057 learned_addr learned my addr 172.21.15.64:0/374071057 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.637+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d840580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d667c6640 1 -- 172.21.15.64:0/374071057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 msgr2=0x55780d840580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d667c6640 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d840580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d667c6640 1 -- 172.21.15.64:0/374071057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 msgr2=0x55780d84f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d667c6640 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d667c6640 1 -- 172.21.15.64:0/374071057 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55780d517ba0 con 0x55780d84f8b0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d667c6640 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d851ca0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8d580cd3a0 tx=0x7f8d5809bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da915a128f1ef188 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:36.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d4bfff640 1 -- 172.21.15.64:0/374071057 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5809c170 con 0x55780d84f8b0 2024-08-19T01:49:36.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d4bfff640 1 -- 172.21.15.64:0/374071057 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d580a0520 con 0x55780d84f8b0 2024-08-19T01:49:36.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.638+0000 7f8d4bfff640 1 -- 172.21.15.64:0/374071057 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58098970 con 0x55780d84f8b0 2024-08-19T01:49:36.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 -- 172.21.15.64:0/374071057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 msgr2=0x55780d851ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:36.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d851ca0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8d580cd3a0 tx=0x7f8d5809bcb0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 -- 172.21.15.64:0/374071057 shutdown_connections 2024-08-19T01:49:36.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d851ca0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 --2- 172.21.15.64:0/374071057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d840580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 -- 172.21.15.64:0/374071057 >> 172.21.15.64:0/374071057 conn(0x55780d83c850 msgr2=0x55780d83cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 -- 172.21.15.64:0/374071057 shutdown_connections 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 -- 172.21.15.64:0/374071057 wait complete. 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.639+0000 7f8d6a627740 1 -- start start 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 1 Processor -- start 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d84c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d84d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55780d852480 con 0x55780d8401a0 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55780d7609a0 con 0x55780d840ac0 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55780d852600 con 0x55780d84f8b0 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d6a627740 10 client.0 _ll_register_callbacks cb 0x55780d74ec90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:49:36.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d65fc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d84c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d657c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d657c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:57772/0 (socket says 172.21.15.64:57772) 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d65fc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d84c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45700/0 (socket says 172.21.15.64:45700) 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d657c4640 1 -- 172.21.15.64:0/3801679638 learned_addr learned my addr 172.21.15.64:0/3801679638 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.640+0000 7f8d667c6640 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d84d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d65fc5640 1 -- 172.21.15.64:0/3801679638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 msgr2=0x55780d84d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d84d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d65fc5640 1 -- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 msgr2=0x55780d84cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d65fc5640 1 -- 172.21.15.64:0/3801679638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55780d76c220 con 0x55780d8401a0 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d657c4640 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d84c7f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0045c0 tx=0x7f8d5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed59dea618ed1395 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d497fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d497fa640 1 -- 172.21.15.64:0/3801679638 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0139f0 con 0x55780d8401a0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d580a0520 con 0x55780d8401a0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d497fa640 1 -- 172.21.15.64:0/3801679638 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7f8d5c013b90 con 0x55780d8401a0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d497fa640 1 -- 172.21.15.64:0/3801679638 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8d5c013d90 con 0x55780d8401a0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d6a627740 10 client.16158 fetch_fsmap learned FSMap version 32 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55780d85e270 con 0x55780d8401a0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.641+0000 7f8d497fa640 1 -- 172.21.15.64:0/3801679638 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d580a0520 con 0x55780d8401a0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d497fa640 1 -- 172.21.15.64:0/3801679638 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f8d5c029500 con 0x55780d8401a0 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d6a627740 10 client.16158 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d6a627740 10 client.16158 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d6a627740 20 client.16158 populate_metadata read hostname 'smithi064' 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d6a627740 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d85f900 0x55780d861cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d6a627740 4 client.16158 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:49:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55780d7b2f40 con 0x55780d85f900 2024-08-19T01:49:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.642+0000 7f8d657c4640 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d85f900 0x55780d861cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.643+0000 7f8d657c4640 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d85f900 0x55780d861cf0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb0cb49912194c48 server_cookie=b386c962de06a4f5 in_seq=0 out_seq=0 2024-08-19T01:49:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.643+0000 7f8d497fa640 10 client.16158 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.645+0000 7f8d497fa640 1 -- 172.21.15.64:0/3801679638 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55780d7b2f40 con 0x55780d85f900 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.645+0000 7f8d497fa640 10 client.16158 handle_command_reply: tid=0 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.645+0000 7f8d6a627740 1 client.16158 shutdown 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.645+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 msgr2=0x55780d84c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.645+0000 7f8d6a627740 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d84c7f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0045c0 tx=0x7f8d5c004730 comp rx=0 tx=0).stop 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 shutdown_connections 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d84d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d85f900 0x55780d861cf0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d84cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 --2- 172.21.15.64:0/3801679638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d84c7f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d497fa640 0 client.16158 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:49:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 >> 172.21.15.64:0/3801679638 conn(0x55780d83c850 msgr2=0x55780d83bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 shutdown_connections 2024-08-19T01:49:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 1 -- 172.21.15.64:0/3801679638 wait complete. 2024-08-19T01:49:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.646+0000 7f8d6a627740 20 client.16158 trim_cache size 0 max 16384 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 Processor -- start 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 -- start start 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d83ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d867380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d8678c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8d601af5a0 con 0x55780d8401a0 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d601ace70 con 0x55780d840ac0 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d6a627740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8d601aace0 con 0x55780d84f8b0 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d65fc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d83ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d65fc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d83ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45706/0 (socket says 172.21.15.64:45706) 2024-08-19T01:49:36.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d65fc5640 1 -- 172.21.15.64:0/375407574 learned_addr learned my addr 172.21.15.64:0/375407574 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d657c4640 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d867380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.667+0000 7f8d667c6640 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d8678c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d65fc5640 1 -- 172.21.15.64:0/375407574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 msgr2=0x55780d8678c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d8678c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d65fc5640 1 -- 172.21.15.64:0/375407574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 msgr2=0x55780d867380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d867380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d65fc5640 1 -- 172.21.15.64:0/375407574 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55780d848350 con 0x55780d8401a0 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d83ba20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009a60 tx=0x7f8d5c009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d15f90fe708ebc server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d4b7fe640 1 -- 172.21.15.64:0/375407574 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c029d60 con 0x55780d8401a0 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d4b7fe640 1 -- 172.21.15.64:0/375407574 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d5c03b030 con 0x55780d8401a0 2024-08-19T01:49:36.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.668+0000 7f8d4b7fe640 1 -- 172.21.15.64:0/375407574 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c037790 con 0x55780d8401a0 2024-08-19T01:49:36.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 -- 172.21.15.64:0/375407574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 msgr2=0x55780d83ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:36.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d83ba20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009a60 tx=0x7f8d5c009730 comp rx=0 tx=0).stop 2024-08-19T01:49:36.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 -- 172.21.15.64:0/375407574 shutdown_connections 2024-08-19T01:49:36.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d84f8b0 0x55780d8678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d867380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 --2- 172.21.15.64:0/375407574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d8401a0 0x55780d83ba20 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 -- 172.21.15.64:0/375407574 >> 172.21.15.64:0/375407574 conn(0x55780d83c850 msgr2=0x55780d83b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:36.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 -- 172.21.15.64:0/375407574 shutdown_connections 2024-08-19T01:49:36.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 -- 172.21.15.64:0/375407574 wait complete. 2024-08-19T01:49:36.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 -- start start 2024-08-19T01:49:36.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.669+0000 7f8d6a627740 1 Processor -- start 2024-08-19T01:49:36.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d857f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d853290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d6a627740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d8537d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d6a627740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8d601ab330 con 0x55780d84f8b0 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d6a627740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d601ac7a0 con 0x55780d840ac0 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d6a627740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8d601aba00 con 0x55780d8401a0 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d6a627740 10 client.0 _ll_register_callbacks cb 0x7f8d5c011d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d65fc5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d857f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d657c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d853290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d65fc5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d857f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:55724/0 (socket says 172.21.15.64:55724) 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d65fc5640 1 -- 172.21.15.64:0/2472012370 learned_addr learned my addr 172.21.15.64:0/2472012370 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d667c6640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d8537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d657c4640 1 -- 172.21.15.64:0/2472012370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 msgr2=0x55780d857f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d657c4640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d857f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d657c4640 1 -- 172.21.15.64:0/2472012370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 msgr2=0x55780d8537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:49:36.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d657c4640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d8537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:36.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d657c4640 1 -- 172.21.15.64:0/2472012370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55780d854000 con 0x55780d840ac0 2024-08-19T01:49:36.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d667c6640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d8537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:49:36.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.670+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d857f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:49:36.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.671+0000 7f8d657c4640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d853290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8d50006010 tx=0x7f8d500051f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7502578585785d5b server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:49:36.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.671+0000 7f8d497fa640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.671+0000 7f8d497fa640 1 -- 172.21.15.64:0/2472012370 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d50035040 con 0x55780d840ac0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.671+0000 7f8d497fa640 1 -- 172.21.15.64:0/2472012370 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7f8d50004760 con 0x55780d840ac0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.671+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d5c03b030 con 0x55780d840ac0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.672+0000 7f8d497fa640 1 -- 172.21.15.64:0/2472012370 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8d50055050 con 0x55780d840ac0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.673+0000 7f8d6a627740 10 client.25565 fetch_fsmap learned FSMap version 32 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.673+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55780d8ca320 con 0x55780d840ac0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d497fa640 1 -- 172.21.15.64:0/2472012370 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d50004e40 con 0x55780d840ac0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d497fa640 1 -- 172.21.15.64:0/2472012370 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f8d500035f0 con 0x55780d840ac0 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d6a627740 10 client.25565 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d6a627740 10 client.25565 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:49:36.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d6a627740 20 client.25565 populate_metadata read hostname 'smithi064' 2024-08-19T01:49:36.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d6a627740 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d8caa90 0x55780d8cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:49:36.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d6a627740 4 client.25565 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}] 2024-08-19T01:49:36.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.674+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}) -- 0x7f8d5c03b030 con 0x55780d8caa90 2024-08-19T01:49:36.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.676+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d8caa90 0x55780d8cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:49:36.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.679+0000 7f8d497fa640 10 client.25565 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:49:36.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:36.679+0000 7f8d65fc5640 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d8caa90 0x55780d8cceb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac61d6ace5255305 server_cookie=dcd9a077b4093db7 in_seq=0 out_seq=0 2024-08-19T01:49:36.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63768 26.31 MB/sec execute 68 sec latency 73.156 ms 2024-08-19T01:49:37.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.478+0000 7f8d497fa640 1 -- 172.21.15.64:0/2472012370 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f8d5c03b030 con 0x55780d8caa90 2024-08-19T01:49:37.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.478+0000 7f8d497fa640 10 client.25565 handle_command_reply: tid=0 2024-08-19T01:49:37.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.478+0000 7f8d6a627740 1 client.25565 shutdown 2024-08-19T01:49:37.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.478+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 msgr2=0x55780d853290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:37.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.478+0000 7f8d6a627740 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d853290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8d50006010 tx=0x7f8d500051f0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 shutdown_connections 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55780d84f8b0 0x55780d8537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55780d8caa90 0x55780d8cceb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55780d840ac0 0x55780d853290 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 --2- 172.21.15.64:0/2472012370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55780d8401a0 0x55780d857f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d497fa640 0 client.25565 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 >> 172.21.15.64:0/2472012370 conn(0x55780d83c850 msgr2=0x55780d83bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 shutdown_connections 2024-08-19T01:49:37.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 1 -- 172.21.15.64:0/2472012370 wait complete. 2024-08-19T01:49:37.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d6a627740 20 client.25565 trim_cache size 0 max 16384 2024-08-19T01:49:37.495 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:49:37.495 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 32, 2024-08-19T01:49:37.495 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:49:37.495 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 58, 2024-08-19T01:49:37.495 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:49:37.495 INFO:teuthology.orchestra.run.smithi064.stdout: "68277463": { 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 58, 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 147.1, 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:49:37.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:49:37.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8d4c0a2cf0 msgr2=0x7f8d4c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8d4c0a2cf0 0x7f8d4c0a51b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009ba0 tx=0x7f8d5c005f90 comp rx=0 tx=0).stop 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d601067b0 msgr2=0x7f8d6019ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d601067b0 0x7f8d6019ca20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800eb10 tx=0x7f8d5800c2d0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 shutdown_connections 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f8d4c0a2cf0 0x7f8d4c0a51b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d601067b0 0x7f8d6019ca20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d60101150 0x7f8d60198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 --2- 172.21.15.64:0/1290440316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d601007b0 0x7f8d60198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 >> 172.21.15.64:0/1290440316 conn(0x7f8d600fc4d0 msgr2=0x7f8d6006a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:49:37.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 shutdown_connections 2024-08-19T01:49:37.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:49:37.479+0000 7f8d68229640 1 -- 172.21.15.64:0/1290440316 wait complete. 2024-08-19T01:49:37.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 25.95 MB/sec execute 69 sec latency 1057.255 ms 2024-08-19T01:49:38.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:37 smithi064 ceph-mon[18515]: pgmap v514: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 599 op/s 2024-08-19T01:49:38.118 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '68277463', quiesce took 4.7 seconds. Will release after: 22.35 2024-08-19T01:49:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:37 smithi123 ceph-mon[21829]: pgmap v514: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 599 op/s 2024-08-19T01:49:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:37 smithi161 ceph-mon[21757]: pgmap v514: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 599 op/s 2024-08-19T01:49:38.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 25.58 MB/sec execute 70 sec latency 2057.308 ms 2024-08-19T01:49:39.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:38 smithi123 ceph-mon[21829]: pgmap v515: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.3 MiB/s wr, 375 op/s 2024-08-19T01:49:39.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:38 smithi161 ceph-mon[21757]: pgmap v515: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.3 MiB/s wr, 375 op/s 2024-08-19T01:49:39.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:38 smithi064 ceph-mon[18515]: pgmap v515: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.3 MiB/s wr, 375 op/s 2024-08-19T01:49:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 25.22 MB/sec execute 71 sec latency 3057.346 ms 2024-08-19T01:49:40.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 24.87 MB/sec execute 72 sec latency 4057.391 ms 2024-08-19T01:49:41.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:41 smithi123 ceph-mon[21829]: pgmap v516: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 8.7 MiB/s wr, 495 op/s 2024-08-19T01:49:41.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:41 smithi161 ceph-mon[21757]: pgmap v516: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 8.7 MiB/s wr, 495 op/s 2024-08-19T01:49:41.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:41 smithi064 ceph-mon[18515]: pgmap v516: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 8.7 MiB/s wr, 495 op/s 2024-08-19T01:49:41.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 24.52 MB/sec execute 73 sec latency 5057.439 ms 2024-08-19T01:49:42.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 24.19 MB/sec execute 74 sec latency 6057.476 ms 2024-08-19T01:49:43.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:43 smithi123 ceph-mon[21829]: pgmap v517: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 295 op/s 2024-08-19T01:49:43.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:43 smithi161 ceph-mon[21757]: pgmap v517: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 295 op/s 2024-08-19T01:49:43.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:43 smithi064 ceph-mon[18515]: pgmap v517: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 295 op/s 2024-08-19T01:49:43.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 23.87 MB/sec execute 75 sec latency 7057.566 ms 2024-08-19T01:49:44.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 23.56 MB/sec execute 76 sec latency 8057.635 ms 2024-08-19T01:49:45.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:45 smithi123 ceph-mon[21829]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 295 op/s 2024-08-19T01:49:45.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:45 smithi161 ceph-mon[21757]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 295 op/s 2024-08-19T01:49:45.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:45 smithi064 ceph-mon[18515]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 295 op/s 2024-08-19T01:49:45.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 23.25 MB/sec execute 77 sec latency 9057.684 ms 2024-08-19T01:49:46.727 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 22.95 MB/sec execute 78 sec latency 10057.727 ms 2024-08-19T01:49:47.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:47 smithi123 ceph-mon[21829]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.7 MiB/s wr, 350 op/s 2024-08-19T01:49:47.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:47 smithi161 ceph-mon[21757]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.7 MiB/s wr, 350 op/s 2024-08-19T01:49:47.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:47 smithi064 ceph-mon[18515]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.7 MiB/s wr, 350 op/s 2024-08-19T01:49:47.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 22.66 MB/sec execute 79 sec latency 11057.769 ms 2024-08-19T01:49:48.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 22.38 MB/sec execute 80 sec latency 12057.860 ms 2024-08-19T01:49:49.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:49 smithi123 ceph-mon[21829]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 174 op/s 2024-08-19T01:49:49.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:49 smithi161 ceph-mon[21757]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 174 op/s 2024-08-19T01:49:49.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:49 smithi064 ceph-mon[18515]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 174 op/s 2024-08-19T01:49:49.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 22.10 MB/sec execute 81 sec latency 13057.965 ms 2024-08-19T01:49:50.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 21.83 MB/sec execute 82 sec latency 14058.023 ms 2024-08-19T01:49:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:51 smithi123 ceph-mon[21829]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 239 op/s 2024-08-19T01:49:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:51 smithi161 ceph-mon[21757]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 239 op/s 2024-08-19T01:49:51.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:51 smithi064 ceph-mon[18515]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 239 op/s 2024-08-19T01:49:51.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 21.57 MB/sec execute 83 sec latency 15058.080 ms 2024-08-19T01:49:52.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 21.31 MB/sec execute 84 sec latency 16058.132 ms 2024-08-19T01:49:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:53 smithi123 ceph-mon[21829]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 3.6 MiB/s wr, 118 op/s 2024-08-19T01:49:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:53 smithi161 ceph-mon[21757]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 3.6 MiB/s wr, 118 op/s 2024-08-19T01:49:53.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:53 smithi064 ceph-mon[18515]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 3.6 MiB/s wr, 118 op/s 2024-08-19T01:49:53.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 21.06 MB/sec execute 85 sec latency 17058.241 ms 2024-08-19T01:49:54.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 20.82 MB/sec execute 86 sec latency 18058.317 ms 2024-08-19T01:49:55.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:55 smithi123 ceph-mon[21829]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 3.6 MiB/s wr, 118 op/s 2024-08-19T01:49:55.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:55 smithi161 ceph-mon[21757]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 3.6 MiB/s wr, 118 op/s 2024-08-19T01:49:55.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:55 smithi064 ceph-mon[18515]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 3.6 MiB/s wr, 118 op/s 2024-08-19T01:49:55.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 20.58 MB/sec execute 87 sec latency 19058.360 ms 2024-08-19T01:49:56.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 20.34 MB/sec execute 88 sec latency 20058.406 ms 2024-08-19T01:49:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:57 smithi123 ceph-mon[21829]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 5.0 MiB/s wr, 173 op/s 2024-08-19T01:49:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:57 smithi161 ceph-mon[21757]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 5.0 MiB/s wr, 173 op/s 2024-08-19T01:49:57.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:57 smithi064 ceph-mon[18515]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 5.0 MiB/s wr, 173 op/s 2024-08-19T01:49:57.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 20.12 MB/sec execute 89 sec latency 21058.456 ms 2024-08-19T01:49:58.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 19.89 MB/sec execute 90 sec latency 22058.571 ms 2024-08-19T01:49:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:49:59 smithi123 ceph-mon[21829]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 3.2 MiB/s wr, 119 op/s 2024-08-19T01:49:59.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:49:59 smithi161 ceph-mon[21757]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 3.2 MiB/s wr, 119 op/s 2024-08-19T01:49:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:49:59 smithi064 ceph-mon[18515]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 3.2 MiB/s wr, 119 op/s 2024-08-19T01:49:59.728 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 19.67 MB/sec execute 91 sec latency 23058.684 ms 2024-08-19T01:50:00.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:00 smithi161 ceph-mon[21757]: overall HEALTH_OK 2024-08-19T01:50:00.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:50:00.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:00 smithi064 ceph-mon[18515]: overall HEALTH_OK 2024-08-19T01:50:00.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:00 smithi123 ceph-mon[21829]: overall HEALTH_OK 2024-08-19T01:50:00.730 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 19.46 MB/sec execute 92 sec latency 24060.665 ms 2024-08-19T01:50:00.764 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:50:01.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 -- 172.21.15.64:0/2001839719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8107fd0 msgr2=0x7f7ea810e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:01.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2001839719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8107fd0 0x7f7ea810e970 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009930 tx=0x7f7ea002a440 comp rx=0 tx=0).stop 2024-08-19T01:50:01.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 -- 172.21.15.64:0/2001839719 shutdown_connections 2024-08-19T01:50:01.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2001839719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8107fd0 0x7f7ea810e970 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2001839719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea8103410 0x7f7ea81078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2001839719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea8102a60 0x7f7ea8102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 -- 172.21.15.64:0/2001839719 >> 172.21.15.64:0/2001839719 conn(0x7f7ea80fc500 msgr2=0x7f7ea80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 -- 172.21.15.64:0/2001839719 shutdown_connections 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 -- 172.21.15.64:0/2001839719 wait complete. 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 Processor -- start 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.359+0000 7f7eaf381640 1 -- start start 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8102a60 0x7f7ea8198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea8103410 0x7f7ea8198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea8107fd0 0x7f7ea819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea8111e10 con 0x7f7ea8102a60 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea8111c90 con 0x7f7ea8103410 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea8111f90 con 0x7f7ea8107fd0 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8102a60 0x7f7ea8198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8102a60 0x7f7ea8198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33274/0 (socket says 172.21.15.64:33274) 2024-08-19T01:50:01.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 -- 172.21.15.64:0/2795500929 learned_addr learned my addr 172.21.15.64:0/2795500929 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead91e640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea8107fd0 0x7f7ea819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eac91c640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea8103410 0x7f7ea8198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 -- 172.21.15.64:0/2795500929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea8107fd0 msgr2=0x7f7ea819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea8107fd0 0x7f7ea819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 -- 172.21.15.64:0/2795500929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea8103410 msgr2=0x7f7ea8198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea8103410 0x7f7ea8198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 -- 172.21.15.64:0/2795500929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea819d300 con 0x7f7ea8102a60 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ead11d640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8102a60 0x7f7ea8198350 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800eab0 tx=0x7f7e9800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ea67fc640 1 -- 172.21.15.64:0/2795500929 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800ce40 con 0x7f7ea8102a60 2024-08-19T01:50:01.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ea67fc640 1 -- 172.21.15.64:0/2795500929 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e98011950 con 0x7f7ea8102a60 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea819d5f0 con 0x7f7ea8102a60 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7ea67fc640 1 -- 172.21.15.64:0/2795500929 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e98010ae0 con 0x7f7ea8102a60 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.360+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea81a4ed0 con 0x7f7ea8102a60 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.362+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea8102e40 con 0x7f7ea8102a60 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.363+0000 7f7ea67fc640 1 -- 172.21.15.64:0/2795500929 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7e98011b00 con 0x7f7ea8102a60 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.363+0000 7f7ea67fc640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f7e900a2d70 0x7f7e900a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.363+0000 7f7eac91c640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f7e900a2d70 0x7f7e900a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.363+0000 7f7ea67fc640 1 -- 172.21.15.64:0/2795500929 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7f7e980cff60 con 0x7f7ea8102a60 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.364+0000 7f7eac91c640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f7e900a2d70 0x7f7e900a5230 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ea80fe980 tx=0x7f7e9c008040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:01.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.367+0000 7f7ea67fc640 1 -- 172.21.15.64:0/2795500929 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f7e98016030 con 0x7f7ea8102a60 2024-08-19T01:50:01.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:01 smithi161 ceph-mon[21757]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 3.5 MiB/s wr, 144 op/s 2024-08-19T01:50:01.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:01 smithi064 ceph-mon[18515]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 3.5 MiB/s wr, 144 op/s 2024-08-19T01:50:01.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.578+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ea81996b0 con 0x7f7ea8102a60 2024-08-19T01:50:01.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.579+0000 7f7ea67fc640 1 -- 172.21.15.64:0/2795500929 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11921 (secure 0 0 0) 0x7f7e9809c7f0 con 0x7f7ea8102a60 2024-08-19T01:50:01.580 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:50:01.580 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":32,"btime":"2024-08-19T01:49:07:146032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:49:06.572445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:50:01.580 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 32 2024-08-19T01:50:01.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f7e900a2d70 msgr2=0x7f7e900a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:01.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f7e900a2d70 0x7f7e900a5230 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ea80fe980 tx=0x7f7e9c008040 comp rx=0 tx=0).stop 2024-08-19T01:50:01.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8102a60 msgr2=0x7f7ea8198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:01.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8102a60 0x7f7ea8198350 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800eab0 tx=0x7f7e9800ef70 comp rx=0 tx=0).stop 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 shutdown_connections 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f7e900a2d70 0x7f7e900a5230 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea8107fd0 0x7f7ea819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea8103410 0x7f7ea8198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 --2- 172.21.15.64:0/2795500929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea8102a60 0x7f7ea8198350 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 >> 172.21.15.64:0/2795500929 conn(0x7f7ea80fc500 msgr2=0x7f7ea81060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 shutdown_connections 2024-08-19T01:50:01.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:01.581+0000 7f7eaf381640 1 -- 172.21.15.64:0/2795500929 wait complete. 2024-08-19T01:50:01.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:01 smithi123 ceph-mon[21829]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 3.5 MiB/s wr, 144 op/s 2024-08-19T01:50:01.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 19.25 MB/sec execute 93 sec latency 25060.764 ms 2024-08-19T01:50:02.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:02 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2795500929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:02.397 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 68277463 --release --await' 2024-08-19T01:50:02.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 68277463 --release --await 2024-08-19T01:50:02.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:02 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2795500929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:02.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:02 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2795500929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:02.698 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:50:02.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63784 19.05 MB/sec execute 94 sec latency 26060.808 ms 2024-08-19T01:50:03.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.227+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2619318364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb410bea0 msgr2=0x7fedb410e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.227+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2619318364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb410bea0 0x7fedb410e340 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fedac009ca0 tx=0x7fedac02a950 comp rx=0 tx=0).stop 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2619318364 shutdown_connections 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2619318364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb410bea0 0x7fedb410e340 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2619318364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedb4103390 0x7fedb410b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2619318364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedb4102a70 0x7fedb4102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2619318364 >> 172.21.15.64:0/2619318364 conn(0x7fedb40fc4d0 msgr2=0x7fedb40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2619318364 shutdown_connections 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2619318364 wait complete. 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 Processor -- start 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 -- start start 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4102a70 0x7fedb4198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.228+0000 7fedbb4b1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedb4103390 0x7fedb4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedbb4b1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedb410bea0 0x7fedb419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedbb4b1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedb4111c80 con 0x7fedb4102a70 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedbb4b1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fedb4111b00 con 0x7fedb4103390 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedbb4b1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fedb4111e00 con 0x7fedb410bea0 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4102a70 0x7fedb4198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4102a70 0x7fedb4198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33296/0 (socket says 172.21.15.64:33296) 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 -- 172.21.15.64:0/2254704883 learned_addr learned my addr 172.21.15.64:0/2254704883 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedb410bea0 0x7fedb419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedb4103390 0x7fedb4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 -- 172.21.15.64:0/2254704883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedb410bea0 msgr2=0x7fedb419ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedb410bea0 0x7fedb419ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 -- 172.21.15.64:0/2254704883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedb4103390 msgr2=0x7fedb4198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedb4103390 0x7fedb4198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 -- 172.21.15.64:0/2254704883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb419d0f0 con 0x7fedb4102a70 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedb924d640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4102a70 0x7fedb4198140 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fedb000d950 tx=0x7fedb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedaa7fc640 1 -- 172.21.15.64:0/2254704883 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0017070 con 0x7fedb4102a70 2024-08-19T01:50:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedaa7fc640 1 -- 172.21.15.64:0/2254704883 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb0009c30 con 0x7fedb4102a70 2024-08-19T01:50:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb419d3e0 con 0x7fedb4102a70 2024-08-19T01:50:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedaa7fc640 1 -- 172.21.15.64:0/2254704883 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb000f6b0 con 0x7fedb4102a70 2024-08-19T01:50:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.229+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb41a4cc0 con 0x7fedb4102a70 2024-08-19T01:50:03.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.232+0000 7fedaa7fc640 1 -- 172.21.15.64:0/2254704883 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fedb00109d0 con 0x7fedb4102a70 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.232+0000 7fedbd8af740 1 Processor -- start 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.232+0000 7fedbd8af740 1 -- start start 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0d4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0df630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0e1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedbd8af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56228f03ac60 con 0x56228f0dfb70 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedbd8af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56228efd2b90 con 0x56228f0d4990 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedbd8af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56228f036c20 con 0x56228f0d52b0 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedaa7fc640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feda00a2d70 0x7feda00a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb924d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0df630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb9a4e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0d4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb8a4c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0e1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb9a4e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0d4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:42634/0 (socket says 172.21.15.64:42634) 2024-08-19T01:50:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb9a4e640 1 -- 172.21.15.64:0/3242938113 learned_addr learned my addr 172.21.15.64:0/3242938113 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb924d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0df630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:42664/0 (socket says 172.21.15.64:42664) 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedaa7fc640 1 -- 172.21.15.64:0/2254704883 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7fedb00cf500 con 0x7fedb4102a70 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feda00a2d70 0x7feda00a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb924d640 1 -- 172.21.15.64:0/3242938113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 msgr2=0x56228f0d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb924d640 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0d4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb924d640 1 -- 172.21.15.64:0/3242938113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 msgr2=0x56228f0e1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb924d640 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0e1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb924d640 1 -- 172.21.15.64:0/3242938113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56228ed4a580 con 0x56228f0d52b0 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.233+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0e1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.234+0000 7fedb924d640 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0df630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fedb00cc840 tx=0x7fedb0066630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=284ee942082cdfb9 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.234+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feda00a2d70 0x7feda00a5230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fedac005fa0 tx=0x7fedac02a950 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.234+0000 7fed9f7fe640 1 -- 172.21.15.64:0/3242938113 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0020040 con 0x56228f0d52b0 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.234+0000 7fed9f7fe640 1 -- 172.21.15.64:0/3242938113 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb009bd10 con 0x56228f0d52b0 2024-08-19T01:50:03.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.234+0000 7fed9f7fe640 1 -- 172.21.15.64:0/3242938113 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb00a4280 con 0x56228f0d52b0 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.234+0000 7fedbd8af740 1 -- 172.21.15.64:0/3242938113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 msgr2=0x56228f0df630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.234+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0df630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fedb00cc840 tx=0x7fedb0066630 comp rx=0 tx=0).stop 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 -- 172.21.15.64:0/3242938113 shutdown_connections 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0e1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0df630 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3242938113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0d4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 -- 172.21.15.64:0/3242938113 >> 172.21.15.64:0/3242938113 conn(0x56228f0d0f20 msgr2=0x56228f0d1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 -- 172.21.15.64:0/3242938113 shutdown_connections 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 -- 172.21.15.64:0/3242938113 wait complete. 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 -- start start 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.235+0000 7fedbd8af740 1 Processor -- start 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d4990 0x56228f0dcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0dd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0dfb70 0x56228f0dd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56228f0e2740 con 0x56228f0d4990 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56228efd87c0 con 0x56228f0dfb70 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56228f0e28c0 con 0x56228f0d52b0 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 10 client.0 _ll_register_callbacks cb 0x56228efe1f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d4990 0x56228f0dcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d4990 0x56228f0dcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33312/0 (socket says 172.21.15.64:33312) 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 -- 172.21.15.64:0/1379184696 learned_addr learned my addr 172.21.15.64:0/1379184696 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0dfb70 0x56228f0dd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 -- 172.21.15.64:0/1379184696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 msgr2=0x56228f0dd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0dd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 -- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0dfb70 msgr2=0x56228f0dd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0dfb70 0x56228f0dd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 -- 172.21.15.64:0/1379184696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56228ee14960 con 0x56228f0d4990 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0dfb70 0x56228f0dd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedb924d640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d4990 0x56228f0dcb00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fedb0097a70 tx=0x7fedb000dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4aeb5f0c19e0341 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fed9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fed9cff9640 1 -- 172.21.15.64:0/1379184696 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0020040 con 0x56228f0d4990 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fed9cff9640 1 -- 172.21.15.64:0/1379184696 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7fedb00cf1b0 con 0x56228f0d4990 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.236+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedb009bd10 con 0x56228f0d4990 2024-08-19T01:50:03.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fed9cff9640 1 -- 172.21.15.64:0/1379184696 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fedb009b610 con 0x56228f0d4990 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 10 client.16182 fetch_fsmap learned FSMap version 32 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56228f0ee820 con 0x56228f0d4990 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fed9cff9640 1 -- 172.21.15.64:0/1379184696 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb009bd10 con 0x56228f0d4990 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fed9cff9640 1 -- 172.21.15.64:0/1379184696 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fedb00a3df0 con 0x56228f0d4990 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 10 client.16182 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 10 client.16182 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 20 client.16182 populate_metadata read hostname 'smithi064' 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f0efb70 0x56228f0f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 4 client.16182 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56228f0481d0 con 0x56228f0efb70 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.237+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f0efb70 0x56228f0f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.238+0000 7fed9cff9640 10 client.16182 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.238+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f0efb70 0x56228f0f1f60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1abc36688a3f90c8 server_cookie=2c8827b10babbd24 in_seq=0 out_seq=0 2024-08-19T01:50:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.238+0000 7fed9cff9640 1 -- 172.21.15.64:0/1379184696 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56228f0481d0 con 0x56228f0efb70 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.238+0000 7fed9cff9640 10 client.16182 handle_command_reply: tid=0 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 client.16182 shutdown 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d4990 msgr2=0x56228f0dcb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d4990 0x56228f0dcb00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fedb0097a70 tx=0x7fedb000dfa0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 shutdown_connections 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f0efb70 0x56228f0f1f60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0dfb70 0x56228f0dd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d52b0 0x56228f0dd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 --2- 172.21.15.64:0/1379184696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d4990 0x56228f0dcb00 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fed9cff9640 0 client.16182 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 >> 172.21.15.64:0/1379184696 conn(0x56228f0d0f20 msgr2=0x56228f0d6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 shutdown_connections 2024-08-19T01:50:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 1 -- 172.21.15.64:0/1379184696 wait complete. 2024-08-19T01:50:03.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.239+0000 7fedbd8af740 20 client.16182 trim_cache size 0 max 16384 2024-08-19T01:50:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 Processor -- start 2024-08-19T01:50:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 -- start start 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d4990 0x56228f0dea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d52b0 0x56228f0f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41b1890 con 0x56228f0dfb70 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41af160 con 0x56228f0d52b0 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedbd8af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41acfd0 con 0x56228f0d4990 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.256+0000 7fedb9a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0f54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33318/0 (socket says 172.21.15.64:33318) 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 -- 172.21.15.64:0/2215098752 learned_addr learned my addr 172.21.15.64:0/2215098752 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb924d640 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d4990 0x56228f0dea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d52b0 0x56228f0f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 -- 172.21.15.64:0/2215098752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d4990 msgr2=0x56228f0dea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d4990 0x56228f0dea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 -- 172.21.15.64:0/2215098752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d52b0 msgr2=0x56228f0f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d52b0 0x56228f0f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 -- 172.21.15.64:0/2215098752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56228f0db050 con 0x56228f0dfb70 2024-08-19T01:50:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0f54c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fedac02fae0 tx=0x7fedac042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60cbfc3e5d426b5 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:03.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fed9effd640 1 -- 172.21.15.64:0/2215098752 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac050070 con 0x56228f0dfb70 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fed9effd640 1 -- 172.21.15.64:0/2215098752 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedac003b70 con 0x56228f0dfb70 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.257+0000 7fed9effd640 1 -- 172.21.15.64:0/2215098752 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac034040 con 0x56228f0dfb70 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 -- 172.21.15.64:0/2215098752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 msgr2=0x56228f0f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0f54c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fedac02fae0 tx=0x7fedac042000 comp rx=0 tx=0).stop 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 -- 172.21.15.64:0/2215098752 shutdown_connections 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0dfb70 0x56228f0f54c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d52b0 0x56228f0f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 --2- 172.21.15.64:0/2215098752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0d4990 0x56228f0dea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 -- 172.21.15.64:0/2215098752 >> 172.21.15.64:0/2215098752 conn(0x56228f0d0f20 msgr2=0x56228f0de360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 -- 172.21.15.64:0/2215098752 shutdown_connections 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 -- 172.21.15.64:0/2215098752 wait complete. 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 -- start start 2024-08-19T01:50:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.258+0000 7fedbd8af740 1 Processor -- start 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0f5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d52b0 0x56228f0f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedbd8af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0dfb70 0x56228f0f3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedbd8af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41ad620 con 0x56228f0d52b0 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedbd8af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41aea90 con 0x56228f0d4990 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedbd8af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41adcf0 con 0x56228f0dfb70 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedbd8af740 10 client.0 _ll_register_callbacks cb 0x7fedb009b3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d52b0 0x56228f0f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d52b0 0x56228f0f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33330/0 (socket says 172.21.15.64:33330) 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 -- 172.21.15.64:0/3527720317 learned_addr learned my addr 172.21.15.64:0/3527720317 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb9a4e640 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0dfb70 0x56228f0f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb924d640 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 -- 172.21.15.64:0/3527720317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0dfb70 msgr2=0x56228f0f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0dfb70 0x56228f0f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 -- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 msgr2=0x56228f0f5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0f5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.259+0000 7fedb8a4c640 1 -- 172.21.15.64:0/3527720317 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56228f1594b0 con 0x56228f0d52b0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedb8a4c640 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d52b0 0x56228f0f2b80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7feda4004700 tx=0x7feda4004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb73e34e5cdddf3 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fed9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fed9cff9640 1 -- 172.21.15.64:0/3527720317 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4012450 con 0x56228f0d52b0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fed9cff9640 1 -- 172.21.15.64:0/3527720317 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7feda40125f0 con 0x56228f0d52b0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedac003b70 con 0x56228f0d52b0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fed9cff9640 1 -- 172.21.15.64:0/3527720317 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feda40127f0 con 0x56228f0d52b0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 10 client.16188 fetch_fsmap learned FSMap version 32 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56228f15a6d0 con 0x56228f0d52b0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fed9cff9640 1 -- 172.21.15.64:0/3527720317 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedac003b70 con 0x56228f0d52b0 2024-08-19T01:50:03.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fed9cff9640 1 -- 172.21.15.64:0/3527720317 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7feda40279a0 con 0x56228f0d52b0 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 10 client.16188 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 10 client.16188 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 20 client.16188 populate_metadata read hostname 'smithi064' 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f15ae80 0x56228f15d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 4 client.16188 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "68277463", "release": true, "await": true}] 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.260+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "68277463", "release": true, "await": true}) -- 0x56228f0426e0 con 0x56228f15ae80 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.261+0000 7fedb924d640 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f15ae80 0x56228f15d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.261+0000 7fed9cff9640 10 client.16188 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.261+0000 7fedb924d640 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f15ae80 0x56228f15d2a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cf5f75efcb0e6e2 server_cookie=bbf9426d630e0a28 in_seq=0 out_seq=0 2024-08-19T01:50:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.332+0000 7fed9cff9640 1 -- 172.21.15.64:0/3527720317 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56228f0426e0 con 0x56228f15ae80 2024-08-19T01:50:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.332+0000 7fed9cff9640 10 client.16188 handle_command_reply: tid=0 2024-08-19T01:50:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.332+0000 7fedbd8af740 1 client.16188 shutdown 2024-08-19T01:50:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.332+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d52b0 msgr2=0x56228f0f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.332+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d52b0 0x56228f0f2b80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7feda4004700 tx=0x7feda4004410 comp rx=0 tx=0).stop 2024-08-19T01:50:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 shutdown_connections 2024-08-19T01:50:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56228f0dfb70 0x56228f0f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56228f0d52b0 0x56228f0f2b80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x56228f15ae80 0x56228f15d2a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 --2- 172.21.15.64:0/3527720317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56228f0d4990 0x56228f0f5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fed9cff9640 0 client.16188 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 >> 172.21.15.64:0/3527720317 conn(0x56228f0d0f20 msgr2=0x56228f0eb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 shutdown_connections 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 1 -- 172.21.15.64:0/3527720317 wait complete. 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbd8af740 20 client.16188 trim_cache size 0 max 16384 2024-08-19T01:50:03.335 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 32, 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 60, 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "68277463": { 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 60, 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:50:03.336 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 147.1, 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:03.337 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:50:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feda00a2d70 msgr2=0x7feda00a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feda00a2d70 0x7feda00a5230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fedac005fa0 tx=0x7fedac02a950 comp rx=0 tx=0).stop 2024-08-19T01:50:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4102a70 msgr2=0x7fedb4198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.333+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4102a70 0x7fedb4198140 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fedb000d950 tx=0x7fedb000de10 comp rx=0 tx=0).stop 2024-08-19T01:50:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 shutdown_connections 2024-08-19T01:50:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feda00a2d70 0x7feda00a5230 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fedb410bea0 0x7fedb419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedb4103390 0x7fedb4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 --2- 172.21.15.64:0/2254704883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedb4102a70 0x7fedb4198140 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 >> 172.21.15.64:0/2254704883 conn(0x7fedb40fc4d0 msgr2=0x7fedb40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 shutdown_connections 2024-08-19T01:50:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:03.334+0000 7fedbb4b1640 1 -- 172.21.15.64:0/2254704883 wait complete. 2024-08-19T01:50:03.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:03 smithi161 ceph-mon[21757]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-08-19T01:50:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:03 smithi064 ceph-mon[18515]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-08-19T01:50:03.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:03 smithi123 ceph-mon[21829]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-08-19T01:50:03.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 63926 18.92 MB/sec execute 95 sec latency 26662.956 ms 2024-08-19T01:50:04.088 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '68277463', total seconds elapsed: 30.7 2024-08-19T01:50:04.088 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.05 2024-08-19T01:50:04.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:50:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:50:04.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 64398 18.97 MB/sec execute 96 sec latency 126.933 ms 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 -- 172.21.15.64:0/1765537952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 msgr2=0x7f47c810e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 --2- 172.21.15.64:0/1765537952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 0x7f47c810e790 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009cd0 tx=0x7f47c002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 -- 172.21.15.64:0/1765537952 shutdown_connections 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 --2- 172.21.15.64:0/1765537952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 0x7f47c810e790 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 --2- 172.21.15.64:0/1765537952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47c81018c0 0x7f47c8101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 --2- 172.21.15.64:0/1765537952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 0x7f47c8107ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 -- 172.21.15.64:0/1765537952 >> 172.21.15.64:0/1765537952 conn(0x7f47c80fd620 msgr2=0x7f47c80ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 -- 172.21.15.64:0/1765537952 shutdown_connections 2024-08-19T01:50:04.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.964+0000 7f47ce011640 1 -- 172.21.15.64:0/1765537952 wait complete. 2024-08-19T01:50:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 Processor -- start 2024-08-19T01:50:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 -- start start 2024-08-19T01:50:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47c81018c0 0x7f47c8198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 0x7f47c81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 0x7f47c819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f47c8111cc0 con 0x7f47c8102260 2024-08-19T01:50:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47c8111b40 con 0x7f47c81018c0 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47ce011640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47c8111e40 con 0x7f47c81078c0 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.965+0000 7f47c6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 0x7f47c81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 0x7f47c819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 0x7f47c819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:42716/0 (socket says 172.21.15.64:42716) 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c7fff640 1 -- 172.21.15.64:0/3820978686 learned_addr learned my addr 172.21.15.64:0/3820978686 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c77fe640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47c81018c0 0x7f47c8198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c77fe640 1 -- 172.21.15.64:0/3820978686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 msgr2=0x7f47c819ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c77fe640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 0x7f47c819ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c77fe640 1 -- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 msgr2=0x7f47c81986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c77fe640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 0x7f47c81986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c77fe640 1 -- 172.21.15.64:0/3820978686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c819d110 con 0x7f47c81018c0 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c6ffd640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 0x7f47c81986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c7fff640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 0x7f47c819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:50:04.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.966+0000 7f47c77fe640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47c81018c0 0x7f47c8198160 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f47b800e9f0 tx=0x7f47b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:04.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.967+0000 7f47c4ff9640 1 -- 172.21.15.64:0/3820978686 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b800ccc0 con 0x7f47c81018c0 2024-08-19T01:50:04.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.967+0000 7f47c4ff9640 1 -- 172.21.15.64:0/3820978686 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47b8010910 con 0x7f47c81018c0 2024-08-19T01:50:04.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.967+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c819d400 con 0x7f47c81018c0 2024-08-19T01:50:04.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.967+0000 7f47c4ff9640 1 -- 172.21.15.64:0/3820978686 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8018960 con 0x7f47c81018c0 2024-08-19T01:50:04.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.967+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c81a4ce0 con 0x7f47c81018c0 2024-08-19T01:50:04.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.968+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c8101d20 con 0x7f47c81018c0 2024-08-19T01:50:04.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.969+0000 7f47c4ff9640 1 -- 172.21.15.64:0/3820978686 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f47b8010ac0 con 0x7f47c81018c0 2024-08-19T01:50:04.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.970+0000 7f47c4ff9640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f47b00a2cf0 0x7f47b00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:04.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.970+0000 7f47c6ffd640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f47b00a2cf0 0x7f47b00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:04.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.970+0000 7f47c4ff9640 1 -- 172.21.15.64:0/3820978686 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9554+0+0 (secure 0 0 0) 0x7f47b80cff10 con 0x7f47c81018c0 2024-08-19T01:50:04.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.971+0000 7f47c6ffd640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f47b00a2cf0 0x7f47b00a51b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47c80ffc10 tx=0x7f47bc00b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:04.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:04.975+0000 7f47c4ff9640 1 -- 172.21.15.64:0/3820978686 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f47b809c070 con 0x7f47c81018c0 2024-08-19T01:50:05.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.259+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47c81994c0 con 0x7f47c81018c0 2024-08-19T01:50:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:50:05.264 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":32,"btime":"2024-08-19T01:49:07:146032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:49:06.572445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:50:05.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.263+0000 7f47c4ff9640 1 -- 172.21.15.64:0/3820978686 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11921 (secure 0 0 0) 0x7f47b80a0370 con 0x7f47c81018c0 2024-08-19T01:50:05.264 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 32 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f47b00a2cf0 msgr2=0x7f47b00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f47b00a2cf0 0x7f47b00a51b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47c80ffc10 tx=0x7f47bc00b390 comp rx=0 tx=0).stop 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47c81018c0 msgr2=0x7f47c8198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47c81018c0 0x7f47c8198160 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f47b800e9f0 tx=0x7f47b800eeb0 comp rx=0 tx=0).stop 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 shutdown_connections 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f47b00a2cf0 0x7f47b00a51b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47c81078c0 0x7f47c819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47c8102260 0x7f47c81986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:05.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 --2- 172.21.15.64:0/3820978686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47c81018c0 0x7f47c8198160 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:05.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 >> 172.21.15.64:0/3820978686 conn(0x7f47c80fd620 msgr2=0x7f47c80fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:05.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 shutdown_connections 2024-08-19T01:50:05.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:05.272+0000 7f47ce011640 1 -- 172.21.15.64:0/3820978686 wait complete. 2024-08-19T01:50:05.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:05 smithi161 ceph-mon[21757]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-08-19T01:50:05.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:05 smithi064 ceph-mon[18515]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-08-19T01:50:05.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:05 smithi123 ceph-mon[21829]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-08-19T01:50:05.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 64723 18.93 MB/sec execute 97 sec latency 359.836 ms 2024-08-19T01:50:06.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:06 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3820978686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:06.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:06 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3820978686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:06.565 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0' 2024-08-19T01:50:06.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db / --timeout 90 --expiration 147.05 --await-for 45.0 2024-08-19T01:50:06.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:06 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3820978686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:06.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65178 19.00 MB/sec execute 98 sec latency 152.315 ms 2024-08-19T01:50:06.962 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:50:07.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:07 smithi064 ceph-mon[18515]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.7 MiB/s wr, 262 op/s 2024-08-19T01:50:07.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 -- 172.21.15.64:0/1655185066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 msgr2=0x7ffb0410d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1655185066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb0410d670 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009cd0 tx=0x7ffafc02a8f0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 -- 172.21.15.64:0/1655185066 shutdown_connections 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1655185066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb0410d670 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1655185066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb041007a0 0x7ffb04100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1655185066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb041067a0 0x7ffb04106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 -- 172.21.15.64:0/1655185066 >> 172.21.15.64:0/1655185066 conn(0x7ffb040fc4c0 msgr2=0x7ffb040fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 -- 172.21.15.64:0/1655185066 shutdown_connections 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 -- 172.21.15.64:0/1655185066 wait complete. 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.553+0000 7ffb0c457640 1 Processor -- start 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0c457640 1 -- start start 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0c457640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb041007a0 0x7ffb04102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0c457640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb041028b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0c457640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb041067a0 0x7ffb04102df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0c457640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04110ba0 con 0x7ffb04101140 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0c457640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04110a20 con 0x7ffb041067a0 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0c457640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04110d20 con 0x7ffb041007a0 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb041028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb041028b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33372/0 (socket says 172.21.15.64:33372) 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0a9f4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb041067a0 0x7ffb04102df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 -- 172.21.15.64:0/1762056235 learned_addr learned my addr 172.21.15.64:0/1762056235 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb041007a0 0x7ffb04102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 -- 172.21.15.64:0/1762056235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb041007a0 msgr2=0x7ffb04102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb041007a0 0x7ffb04102370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 -- 172.21.15.64:0/1762056235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb041067a0 msgr2=0x7ffb04102df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb041067a0 0x7ffb04102df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 -- 172.21.15.64:0/1762056235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb041a6600 con 0x7ffb04101140 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.554+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb041028b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400e9f0 tx=0x7ffaf400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.555+0000 7ffafb7fe640 1 -- 172.21.15.64:0/1762056235 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf400cd80 con 0x7ffb04101140 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.555+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb041a68f0 con 0x7ffb04101140 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.555+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb041a6e80 con 0x7ffb04101140 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.555+0000 7ffafb7fe640 1 -- 172.21.15.64:0/1762056235 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaf4010910 con 0x7ffb04101140 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.555+0000 7ffafb7fe640 1 -- 172.21.15.64:0/1762056235 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40188f0 con 0x7ffb04101140 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.557+0000 7ffafb7fe640 1 -- 172.21.15.64:0/1762056235 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ffaf4010430 con 0x7ffb04101140 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.557+0000 7ffb0e855740 1 Processor -- start 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0e855740 1 -- start start 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e2b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e35f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e364a0 0x55ccc0e38890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0e855740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ccc0d90810 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0e855740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ccc0d37c70 con 0x55ccc0e364a0 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0e855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ccc0d8c7d0 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0a9f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e2b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0a9f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e2b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33378/0 (socket says 172.21.15.64:33378) 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0a9f4640 1 -- 172.21.15.64:0/166178874 learned_addr learned my addr 172.21.15.64:0/166178874 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e35f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffb099f2640 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e364a0 0x55ccc0e38890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.558+0000 7ffafb7fe640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ffaf00a2cf0 0x7ffaf00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffafb7fe640 1 -- 172.21.15.64:0/1762056235 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9642+0+0 (secure 0 0 0) 0x7ffaf40cfb00 con 0x7ffb04101140 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a1f3640 1 -- 172.21.15.64:0/166178874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e364a0 msgr2=0x55ccc0e38890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e364a0 0x55ccc0e38890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a1f3640 1 -- 172.21.15.64:0/166178874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 msgr2=0x55ccc0e2b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e2b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a1f3640 1 -- 172.21.15.64:0/166178874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccc0b02ba0 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e2b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:50:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ffaf00a2cf0 0x7ffaf00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e35f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000bb70 tx=0x7ffb000107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4798623fac5434e4 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ffaf00a2cf0 0x7ffaf00a51b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffafc035000 tx=0x7ffafc0095c0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffaf8dea640 1 -- 172.21.15.64:0/166178874 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb00004030 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffaf8dea640 1 -- 172.21.15.64:0/166178874 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb000041d0 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.559+0000 7ffaf8dea640 1 -- 172.21.15.64:0/166178874 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb00011680 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 -- 172.21.15.64:0/166178874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2bbe0 msgr2=0x55ccc0e35f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:07.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e35f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000bb70 tx=0x7ffb000107a0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 -- 172.21.15.64:0/166178874 shutdown_connections 2024-08-19T01:50:07.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e364a0 0x55ccc0e38890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e35f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 --2- 172.21.15.64:0/166178874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e2b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 -- 172.21.15.64:0/166178874 >> 172.21.15.64:0/166178874 conn(0x55ccc0e27850 msgr2=0x55ccc0e27c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 -- 172.21.15.64:0/166178874 shutdown_connections 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.560+0000 7ffb0e855740 1 -- 172.21.15.64:0/166178874 wait complete. 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 -- start start 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 Processor -- start 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e333a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e338e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e33e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ccc0e38fb0 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ccc0d4b9a0 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ccc0e39130 con 0x55ccc0e364a0 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0e855740 10 client.0 _ll_register_callbacks cb 0x55ccc0d39c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:50:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.561+0000 7ffb0a1f3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a1f3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e333a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33382/0 (socket says 172.21.15.64:33382) 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a1f3640 1 -- 172.21.15.64:0/1184789244 learned_addr learned my addr 172.21.15.64:0/1184789244 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a1f3640 1 -- 172.21.15.64:0/1184789244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 msgr2=0x55ccc0e33e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e33e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e33e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a1f3640 1 -- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 msgr2=0x55ccc0e338e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e338e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a1f3640 1 -- 172.21.15.64:0/1184789244 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccc0d57220 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e33e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.562+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e338e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e333a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffb00003dc0 tx=0x7ffb000040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9eeb528e5b724c3 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffaedffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:50:07.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffaedffb640 1 -- 172.21.15.64:0/1184789244 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0002a060 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffaedffb640 1 -- 172.21.15.64:0/1184789244 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9642+0+0 (secure 0 0 0) 0x7ffb0000bd20 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb000041d0 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffaedffb640 1 -- 172.21.15.64:0/1184789244 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffb00031050 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffb0e855740 10 client.16206 fetch_fsmap learned FSMap version 32 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.563+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ccc0e44cd0 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffaedffb640 1 -- 172.21.15.64:0/1184789244 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb000041d0 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffaedffb640 1 -- 172.21.15.64:0/1184789244 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ffb000046c0 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffb0e855740 10 client.16206 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffb0e855740 10 client.16206 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffb0e855740 20 client.16206 populate_metadata read hostname 'smithi064' 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0e460e0 0x55ccc0e484d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffb0e855740 4 client.16206 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccc0d9df40 con 0x55ccc0e460e0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.564+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0e460e0 0x55ccc0e484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.565+0000 7ffaedffb640 10 client.16206 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:07.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.565+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0e460e0 0x55ccc0e484d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b4ad76dafe1faa2 server_cookie=ec18c236ea8e0d36 in_seq=0 out_seq=0 2024-08-19T01:50:07.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.567+0000 7ffaedffb640 1 -- 172.21.15.64:0/1184789244 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ccc0d9df40 con 0x55ccc0e460e0 2024-08-19T01:50:07.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.567+0000 7ffaedffb640 10 client.16206 handle_command_reply: tid=0 2024-08-19T01:50:07.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.567+0000 7ffb0e855740 1 client.16206 shutdown 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.567+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 msgr2=0x55ccc0e333a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.567+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e333a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffb00003dc0 tx=0x7ffb000040f0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 shutdown_connections 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e33e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0e460e0 0x55ccc0e484d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e338e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1184789244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e333a0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 >> 172.21.15.64:0/1184789244 conn(0x55ccc0e27850 msgr2=0x55ccc0e26b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:07.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffaedffb640 0 client.16206 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:07.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 shutdown_connections 2024-08-19T01:50:07.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 1 -- 172.21.15.64:0/1184789244 wait complete. 2024-08-19T01:50:07.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.568+0000 7ffb0e855740 20 client.16206 trim_cache size 0 max 16384 2024-08-19T01:50:07.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.588+0000 7ffb0e855740 1 Processor -- start 2024-08-19T01:50:07.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.588+0000 7ffb0e855740 1 -- start start 2024-08-19T01:50:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e4db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e4e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e364a0 0x55ccc0e30620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0e855740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8012b40 con 0x55ccc0e364a0 2024-08-19T01:50:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0e855740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8010410 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0e855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffae800e280 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb099f2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e4e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0a1f3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e4db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0a9f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e364a0 0x55ccc0e30620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0a9f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e364a0 0x55ccc0e30620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33388/0 (socket says 172.21.15.64:33388) 2024-08-19T01:50:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.589+0000 7ffb0a9f4640 1 -- 172.21.15.64:0/1910780742 learned_addr learned my addr 172.21.15.64:0/1910780742 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffb0a9f4640 1 -- 172.21.15.64:0/1910780742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2b2c0 msgr2=0x55ccc0e4db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1910780742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e4db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffb0a9f4640 1 -- 172.21.15.64:0/1910780742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 msgr2=0x55ccc0e4e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1910780742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e4e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffb0a9f4640 1 -- 172.21.15.64:0/1910780742 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccc0e27680 con 0x55ccc0e364a0 2024-08-19T01:50:07.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1910780742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e364a0 0x55ccc0e30620 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffafc030670 tx=0x7ffafc044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc59477a217b76c server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:07.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffaeffff640 1 -- 172.21.15.64:0/1910780742 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc056070 con 0x55ccc0e364a0 2024-08-19T01:50:07.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffaeffff640 1 -- 172.21.15.64:0/1910780742 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffafc036040 con 0x55ccc0e364a0 2024-08-19T01:50:07.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.590+0000 7ffaeffff640 1 -- 172.21.15.64:0/1910780742 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0047c0 con 0x55ccc0e364a0 2024-08-19T01:50:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 -- 172.21.15.64:0/1910780742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e364a0 msgr2=0x55ccc0e30620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1910780742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e364a0 0x55ccc0e30620 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffafc030670 tx=0x7ffafc044000 comp rx=0 tx=0).stop 2024-08-19T01:50:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 -- 172.21.15.64:0/1910780742 shutdown_connections 2024-08-19T01:50:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1910780742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e364a0 0x55ccc0e30620 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1910780742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e4e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1910780742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e4db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 -- 172.21.15.64:0/1910780742 >> 172.21.15.64:0/1910780742 conn(0x55ccc0e27850 msgr2=0x55ccc0e32490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 -- 172.21.15.64:0/1910780742 shutdown_connections 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 -- 172.21.15.64:0/1910780742 wait complete. 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 -- start start 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.591+0000 7ffb0e855740 1 Processor -- start 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e3e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e2feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0e855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e303f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0e855740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffae800e8d0 con 0x55ccc0e2b2c0 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0e855740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffae800fd40 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0e855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffae800efa0 con 0x55ccc0e364a0 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0e855740 10 client.0 _ll_register_callbacks cb 0x7ffafc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e2feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e2feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:42716/0 (socket says 172.21.15.64:42716) 2024-08-19T01:50:07.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 -- 172.21.15.64:0/1389253213 learned_addr learned my addr 172.21.15.64:0/1389253213 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e3e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0a9f4640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e303f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 -- 172.21.15.64:0/1389253213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 msgr2=0x55ccc0e303f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e303f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 -- 172.21.15.64:0/1389253213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 msgr2=0x55ccc0e3e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e3e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb099f2640 1 -- 172.21.15.64:0/1389253213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccc0e30a30 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.592+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e3e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffb099f2640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e2feb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400c6a0 tx=0x7ffaf401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=345a325c49d89d3d server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffaedffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffaedffb640 1 -- 172.21.15.64:0/1389253213 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40982e0 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffafc036040 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffaedffb640 1 -- 172.21.15.64:0/1389253213 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9642+0+0 (secure 0 0 0) 0x7ffaf409b0c0 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffaedffb640 1 -- 172.21.15.64:0/1389253213 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffaf4098480 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffb0e855740 10 client.25610 fetch_fsmap learned FSMap version 32 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ccc0eb09b0 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.593+0000 7ffaedffb640 1 -- 172.21.15.64:0/1389253213 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffafc036040 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffaedffb640 1 -- 172.21.15.64:0/1389253213 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ffaf409ab30 con 0x55ccc0e2bbe0 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0e855740 10 client.25610 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0e855740 10 client.25610 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0e855740 20 client.25610 populate_metadata read hostname 'smithi064' 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0eb11a0 0x55ccc0eb35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0e855740 4 client.25610 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}] 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.05, "await_for": 45.0}) -- 0x55ccc0d98290 con 0x55ccc0eb11a0 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0eb11a0 0x55ccc0eb35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffb0a1f3640 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0eb11a0 0x55ccc0eb35c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=577a7ec078468579 server_cookie=bf222728b9ec1e19 in_seq=0 out_seq=0 2024-08-19T01:50:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:07.594+0000 7ffaedffb640 10 client.25610 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:07.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:07 smithi123 ceph-mon[21829]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.7 MiB/s wr, 262 op/s 2024-08-19T01:50:07.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:07 smithi161 ceph-mon[21757]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.7 MiB/s wr, 262 op/s 2024-08-19T01:50:07.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65642 19.06 MB/sec execute 99 sec latency 135.524 ms 2024-08-19T01:50:08.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffaedffb640 1 -- 172.21.15.64:0/1389253213 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ccc0d98290 con 0x55ccc0eb11a0 2024-08-19T01:50:08.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffaedffb640 10 client.25610 handle_command_reply: tid=0 2024-08-19T01:50:08.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 client.25610 shutdown 2024-08-19T01:50:08.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 msgr2=0x55ccc0e2feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:08.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e2feb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400c6a0 tx=0x7ffaf401f000 comp rx=0 tx=0).stop 2024-08-19T01:50:08.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 shutdown_connections 2024-08-19T01:50:08.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ccc0e364a0 0x55ccc0e303f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55ccc0eb11a0 0x55ccc0eb35c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ccc0e2bbe0 0x55ccc0e2feb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 --2- 172.21.15.64:0/1389253213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ccc0e2b2c0 0x55ccc0e3e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffaedffb640 0 client.25610 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.296+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 >> 172.21.15.64:0/1389253213 conn(0x55ccc0e27850 msgr2=0x55ccc0e31c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 shutdown_connections 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0e855740 1 -- 172.21.15.64:0/1389253213 wait complete. 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0e855740 20 client.25610 trim_cache size 0 max 16384 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 32, 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 62, 2024-08-19T01:50:08.300 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "7ef4964d": { 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 62, 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 147.1, 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:50:08.301 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-08-19T01:50:08.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:50:08.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:08.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:08.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:08.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:08.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:08.302 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ffaf00a2cf0 msgr2=0x7ffaf00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ffaf00a2cf0 0x7ffaf00a51b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffafc035000 tx=0x7ffafc0095c0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 msgr2=0x7ffb041028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb041028b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400e9f0 tx=0x7ffaf400eeb0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 shutdown_connections 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ffaf00a2cf0 0x7ffaf00a51b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb041067a0 0x7ffb04102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb04101140 0x7ffb041028b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 --2- 172.21.15.64:0/1762056235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb041007a0 0x7ffb04102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.297+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 >> 172.21.15.64:0/1762056235 conn(0x7ffb040fc4c0 msgr2=0x7ffb0406a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.298+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 shutdown_connections 2024-08-19T01:50:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:08.298+0000 7ffb0c457640 1 -- 172.21.15.64:0/1762056235 wait complete. 2024-08-19T01:50:08.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:08 smithi064 ceph-mon[18515]: osdmap e115: 12 total, 12 up, 12 in 2024-08-19T01:50:08.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:08 smithi123 ceph-mon[21829]: osdmap e115: 12 total, 12 up, 12 in 2024-08-19T01:50:08.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:08 smithi161 ceph-mon[21757]: osdmap e115: 12 total, 12 up, 12 in 2024-08-19T01:50:08.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 18.87 MB/sec execute 100 sec latency 1135.564 ms 2024-08-19T01:50:08.958 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7ef4964d', quiesce took 4.9 seconds. Will release after: 22.15 2024-08-19T01:50:09.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:09 smithi064 ceph-mon[18515]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 248 op/s 2024-08-19T01:50:09.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:09 smithi123 ceph-mon[21829]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 248 op/s 2024-08-19T01:50:09.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:09 smithi161 ceph-mon[21757]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 248 op/s 2024-08-19T01:50:09.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 18.69 MB/sec execute 101 sec latency 2135.613 ms 2024-08-19T01:50:10.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 18.50 MB/sec execute 102 sec latency 3135.717 ms 2024-08-19T01:50:11.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:11 smithi064 ceph-mon[18515]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.3 MiB/s wr, 291 op/s 2024-08-19T01:50:11.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:11 smithi123 ceph-mon[21829]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.3 MiB/s wr, 291 op/s 2024-08-19T01:50:11.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:11 smithi161 ceph-mon[21757]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.3 MiB/s wr, 291 op/s 2024-08-19T01:50:11.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 18.32 MB/sec execute 103 sec latency 4135.773 ms 2024-08-19T01:50:12.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 18.15 MB/sec execute 104 sec latency 5135.816 ms 2024-08-19T01:50:13.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:13 smithi064 ceph-mon[18515]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 6.8 MiB/s wr, 365 op/s 2024-08-19T01:50:13.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:13 smithi123 ceph-mon[21829]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 6.8 MiB/s wr, 365 op/s 2024-08-19T01:50:13.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:13 smithi161 ceph-mon[21757]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 6.8 MiB/s wr, 365 op/s 2024-08-19T01:50:13.731 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 17.97 MB/sec execute 105 sec latency 6135.897 ms 2024-08-19T01:50:14.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 17.80 MB/sec execute 106 sec latency 7136.019 ms 2024-08-19T01:50:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:14 smithi123 ceph-mon[21829]: osdmap e116: 12 total, 12 up, 12 in 2024-08-19T01:50:15.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:14 smithi161 ceph-mon[21757]: osdmap e116: 12 total, 12 up, 12 in 2024-08-19T01:50:15.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:14 smithi064 ceph-mon[18515]: osdmap e116: 12 total, 12 up, 12 in 2024-08-19T01:50:15.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 17.64 MB/sec execute 107 sec latency 8136.110 ms 2024-08-19T01:50:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:15 smithi123 ceph-mon[21829]: pgmap v535: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 241 op/s 2024-08-19T01:50:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:15 smithi161 ceph-mon[21757]: pgmap v535: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 241 op/s 2024-08-19T01:50:16.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:15 smithi064 ceph-mon[18515]: pgmap v535: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 241 op/s 2024-08-19T01:50:16.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 17.47 MB/sec execute 108 sec latency 9136.147 ms 2024-08-19T01:50:17.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 17.31 MB/sec execute 109 sec latency 10136.248 ms 2024-08-19T01:50:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:17 smithi123 ceph-mon[21829]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 251 op/s 2024-08-19T01:50:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:17 smithi161 ceph-mon[21757]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 251 op/s 2024-08-19T01:50:18.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:17 smithi064 ceph-mon[18515]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 251 op/s 2024-08-19T01:50:18.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 17.16 MB/sec execute 110 sec latency 11136.300 ms 2024-08-19T01:50:19.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 17.00 MB/sec execute 111 sec latency 12136.406 ms 2024-08-19T01:50:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:19 smithi123 ceph-mon[21829]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-08-19T01:50:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:19 smithi161 ceph-mon[21757]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-08-19T01:50:20.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:19 smithi064 ceph-mon[18515]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-08-19T01:50:20.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 16.85 MB/sec execute 112 sec latency 13136.452 ms 2024-08-19T01:50:21.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 16.70 MB/sec execute 113 sec latency 14136.548 ms 2024-08-19T01:50:22.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:21 smithi123 ceph-mon[21829]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-08-19T01:50:22.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:21 smithi161 ceph-mon[21757]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-08-19T01:50:22.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:21 smithi064 ceph-mon[18515]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-08-19T01:50:22.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 16.55 MB/sec execute 114 sec latency 15136.663 ms 2024-08-19T01:50:23.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 16.41 MB/sec execute 115 sec latency 16136.777 ms 2024-08-19T01:50:24.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:23 smithi123 ceph-mon[21829]: pgmap v539: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.5 MiB/s wr, 41 op/s 2024-08-19T01:50:24.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:23 smithi161 ceph-mon[21757]: pgmap v539: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.5 MiB/s wr, 41 op/s 2024-08-19T01:50:24.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:23 smithi064 ceph-mon[18515]: pgmap v539: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.5 MiB/s wr, 41 op/s 2024-08-19T01:50:24.732 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 16.27 MB/sec execute 116 sec latency 17136.921 ms 2024-08-19T01:50:25.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 16.13 MB/sec execute 117 sec latency 18137.038 ms 2024-08-19T01:50:26.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:25 smithi123 ceph-mon[21829]: pgmap v540: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:50:26.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:25 smithi161 ceph-mon[21757]: pgmap v540: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:50:26.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:25 smithi064 ceph-mon[18515]: pgmap v540: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:50:26.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 15.99 MB/sec execute 118 sec latency 19137.162 ms 2024-08-19T01:50:27.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 15.86 MB/sec execute 119 sec latency 20137.296 ms 2024-08-19T01:50:28.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:27 smithi123 ceph-mon[21829]: pgmap v541: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:50:28.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:27 smithi161 ceph-mon[21757]: pgmap v541: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:50:28.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:27 smithi064 ceph-mon[18515]: pgmap v541: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.5 MiB/s wr, 40 op/s 2024-08-19T01:50:28.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 15.73 MB/sec execute 120 sec latency 21137.413 ms 2024-08-19T01:50:29.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 15.60 MB/sec execute 121 sec latency 22137.496 ms 2024-08-19T01:50:30.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:29 smithi123 ceph-mon[21829]: pgmap v542: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 942 KiB/s wr, 15 op/s 2024-08-19T01:50:30.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:29 smithi161 ceph-mon[21757]: pgmap v542: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 942 KiB/s wr, 15 op/s 2024-08-19T01:50:30.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:29 smithi064 ceph-mon[18515]: pgmap v542: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 942 KiB/s wr, 15 op/s 2024-08-19T01:50:30.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 15.47 MB/sec execute 122 sec latency 23137.597 ms 2024-08-19T01:50:31.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:50:31.137 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:30 smithi064 ceph-mon[18515]: pgmap v543: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 976 KiB/s wr, 19 op/s 2024-08-19T01:50:31.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:30 smithi123 ceph-mon[21829]: pgmap v543: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 976 KiB/s wr, 19 op/s 2024-08-19T01:50:31.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:30 smithi161 ceph-mon[21757]: pgmap v543: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 976 KiB/s wr, 19 op/s 2024-08-19T01:50:31.380 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:50:31.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 15.34 MB/sec execute 123 sec latency 24137.673 ms 2024-08-19T01:50:31.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 -- 172.21.15.64:0/1405533410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd818100890 msgr2=0x7fd818100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:31.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 --2- 172.21.15.64:0/1405533410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd818100890 0x7fd818100c70 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd808009900 tx=0x7fd80802a410 comp rx=0 tx=0).stop 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 -- 172.21.15.64:0/1405533410 shutdown_connections 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 --2- 172.21.15.64:0/1405533410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81810e270 0x7fd818110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 --2- 172.21.15.64:0/1405533410 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8181011b0 0x7fd81810dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 --2- 172.21.15.64:0/1405533410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd818100890 0x7fd818100c70 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 -- 172.21.15.64:0/1405533410 >> 172.21.15.64:0/1405533410 conn(0x7fd8180fc500 msgr2=0x7fd8180fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.832+0000 7fd81e832640 1 -- 172.21.15.64:0/1405533410 shutdown_connections 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.833+0000 7fd81e832640 1 -- 172.21.15.64:0/1405533410 wait complete. 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.833+0000 7fd81e832640 1 Processor -- start 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.833+0000 7fd81e832640 1 -- start start 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81e832640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd818100890 0x7fd81819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81e832640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd8181011b0 0x7fd81819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81e832640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81810e270 0x7fd81819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:31.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81e832640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd818113f90 con 0x7fd8181011b0 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81e832640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd818113e10 con 0x7fd81810e270 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81e832640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd818114110 con 0x7fd818100890 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd8181011b0 0x7fd81819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd8181011b0 0x7fd81819a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37292/0 (socket says 172.21.15.64:37292) 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 -- 172.21.15.64:0/2450924474 learned_addr learned my addr 172.21.15.64:0/2450924474 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd817fff640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd818100890 0x7fd81819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81cdcf640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81810e270 0x7fd81819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 -- 172.21.15.64:0/2450924474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd818100890 msgr2=0x7fd81819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd818100890 0x7fd81819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 -- 172.21.15.64:0/2450924474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81810e270 msgr2=0x7fd81819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81810e270 0x7fd81819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 -- 172.21.15.64:0/2450924474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81819f3d0 con 0x7fd8181011b0 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81cdcf640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81810e270 0x7fd81819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:50:31.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd80ffff640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd8181011b0 0x7fd81819a960 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd80000c890 tx=0x7fd80000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd815ffb640 1 -- 172.21.15.64:0/2450924474 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800004220 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81819f6c0 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd815ffb640 1 -- 172.21.15.64:0/2450924474 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8000043c0 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.834+0000 7fd815ffb640 1 -- 172.21.15.64:0/2450924474 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800004a00 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.835+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8181a6fa0 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.836+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81806a540 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.837+0000 7fd815ffb640 1 -- 172.21.15.64:0/2450924474 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd8000079b0 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.837+0000 7fd815ffb640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd8040a2cf0 0x7fd8040a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.838+0000 7fd815ffb640 1 -- 172.21.15.64:0/2450924474 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9554+0+0 (secure 0 0 0) 0x7fd8000cfa80 con 0x7fd8181011b0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.838+0000 7fd817fff640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd8040a2cf0 0x7fd8040a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.838+0000 7fd817fff640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd8040a2cf0 0x7fd8040a51b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8080096c0 tx=0x7fd808031490 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:31.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:31.841+0000 7fd815ffb640 1 -- 172.21.15.64:0/2450924474 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd800014030 con 0x7fd8181011b0 2024-08-19T01:50:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.051+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd818109370 con 0x7fd8181011b0 2024-08-19T01:50:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.052+0000 7fd815ffb640 1 -- 172.21.15.64:0/2450924474 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11921 (secure 0 0 0) 0x7fd80009c230 con 0x7fd8181011b0 2024-08-19T01:50:32.054 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:50:32.054 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":32,"btime":"2024-08-19T01:49:07:146032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:49:06.572445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:50:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 32 2024-08-19T01:50:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd8040a2cf0 msgr2=0x7fd8040a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd8040a2cf0 0x7fd8040a51b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8080096c0 tx=0x7fd808031490 comp rx=0 tx=0).stop 2024-08-19T01:50:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd8181011b0 msgr2=0x7fd81819a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd8181011b0 0x7fd81819a960 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd80000c890 tx=0x7fd80000cd50 comp rx=0 tx=0).stop 2024-08-19T01:50:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 shutdown_connections 2024-08-19T01:50:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd8040a2cf0 0x7fd8040a51b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81810e270 0x7fd81819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:32.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd8181011b0 0x7fd81819a960 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:32.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 --2- 172.21.15.64:0/2450924474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd818100890 0x7fd81819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:32.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 >> 172.21.15.64:0/2450924474 conn(0x7fd8180fc500 msgr2=0x7fd8180fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:32.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 shutdown_connections 2024-08-19T01:50:32.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:32.054+0000 7fd81e832640 1 -- 172.21.15.64:0/2450924474 wait complete. 2024-08-19T01:50:32.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:32 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2450924474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:32.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:32 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2450924474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:32.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:32 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2450924474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:50:32.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65645 15.22 MB/sec execute 124 sec latency 25137.726 ms 2024-08-19T01:50:32.835 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --set-id 7ef4964d --release --await' 2024-08-19T01:50:32.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --set-id 7ef4964d --release --await 2024-08-19T01:50:33.134 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:50:33.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:33 smithi123 ceph-mon[21829]: pgmap v544: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 799 KiB/s wr, 17 op/s 2024-08-19T01:50:33.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:33 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:50:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:33 smithi064 ceph-mon[18515]: pgmap v544: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 799 KiB/s wr, 17 op/s 2024-08-19T01:50:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:33 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:50:33.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:33 smithi161 ceph-mon[21757]: pgmap v544: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 799 KiB/s wr, 17 op/s 2024-08-19T01:50:33.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:33 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:50:33.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.639+0000 7feb1ddb1640 1 -- 172.21.15.64:0/689029976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb18102250 msgr2=0x7feb1810e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/689029976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb18102250 0x7feb1810e780 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7feb10009cd0 tx=0x7feb1002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 -- 172.21.15.64:0/689029976 shutdown_connections 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/689029976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb18102250 0x7feb1810e780 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/689029976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb181018b0 0x7feb18101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/689029976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb181078b0 0x7feb18107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 -- 172.21.15.64:0/689029976 >> 172.21.15.64:0/689029976 conn(0x7feb180fd5f0 msgr2=0x7feb180ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 -- 172.21.15.64:0/689029976 shutdown_connections 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 -- 172.21.15.64:0/689029976 wait complete. 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 Processor -- start 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.640+0000 7feb1ddb1640 1 -- start start 2024-08-19T01:50:33.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb1ddb1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb181018b0 0x7feb18198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb1ddb1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb18102250 0x7feb18198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb1ddb1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb181078b0 0x7feb1819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb1ddb1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb18111cb0 con 0x7feb181018b0 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb1ddb1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb18111b30 con 0x7feb181078b0 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb1ddb1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb18111e30 con 0x7feb18102250 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb181018b0 0x7feb18198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb181018b0 0x7feb18198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37306/0 (socket says 172.21.15.64:37306) 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 -- 172.21.15.64:0/942834451 learned_addr learned my addr 172.21.15.64:0/942834451 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb17fff640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb181078b0 0x7feb1819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb0effd640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb18102250 0x7feb18198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 -- 172.21.15.64:0/942834451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb18102250 msgr2=0x7feb18198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb18102250 0x7feb18198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 -- 172.21.15.64:0/942834451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb181078b0 msgr2=0x7feb1819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb181078b0 0x7feb1819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 -- 172.21.15.64:0/942834451 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1819d0e0 con 0x7feb181018b0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb0effd640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb18102250 0x7feb18198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb177fe640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb181018b0 0x7feb18198130 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7feb0800d950 tx=0x7feb0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb157fa640 1 -- 172.21.15.64:0/942834451 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08017030 con 0x7feb181018b0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1819d3d0 con 0x7feb181018b0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb157fa640 1 -- 172.21.15.64:0/942834451 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb08004020 con 0x7feb181018b0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.641+0000 7feb157fa640 1 -- 172.21.15.64:0/942834451 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080047a0 con 0x7feb181018b0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.642+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb181a4cb0 con 0x7feb181018b0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb157fa640 1 -- 172.21.15.64:0/942834451 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feb080041d0 con 0x7feb181018b0 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 Processor -- start 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 -- start start 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad87990 0x560baad87d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad882b0 0x560baad92630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baad94f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560baacedc60 con 0x560baad92b70 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560baac85b90 con 0x560baad87990 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.644+0000 7feb201af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x560baace9c20 con 0x560baad882b0 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad87990 0x560baad87d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad87990 0x560baad87d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:39352/0 (socket says 172.21.15.64:39352) 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baad94f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 -- 172.21.15.64:0/2092908857 learned_addr learned my addr 172.21.15.64:0/2092908857 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb157fa640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feb040a2cf0 0x7feb040a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb177fe640 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad882b0 0x560baad92630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 -- 172.21.15.64:0/2092908857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad882b0 msgr2=0x560baad92630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad882b0 0x560baad92630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 -- 172.21.15.64:0/2092908857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 msgr2=0x560baad94f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baad94f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 -- 172.21.15.64:0/2092908857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560baa9fd580 con 0x560baad87990 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb0effd640 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baad94f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb157fa640 1 -- 172.21.15.64:0/942834451 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9554+0+0 (secure 0 0 0) 0x7feb080cf1f0 con 0x7feb181018b0 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feb040a2cf0 0x7feb040a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb17fff640 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad87990 0x560baad87d70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feb10035000 tx=0x7feb10031660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58f160f03a6278da server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.645+0000 7feb0dffb640 1 -- 172.21.15.64:0/2092908857 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10041040 con 0x560baad87990 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb0dffb640 1 -- 172.21.15.64:0/2092908857 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb10043070 con 0x560baad87990 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb0dffb640 1 -- 172.21.15.64:0/2092908857 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10030d70 con 0x560baad87990 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 -- 172.21.15.64:0/2092908857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad87990 msgr2=0x560baad87d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad87990 0x560baad87d70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feb10035000 tx=0x7feb10031660 comp rx=0 tx=0).stop 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb17fff640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feb040a2cf0 0x7feb040a51b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feb1004c000 tx=0x7feb1004d000 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:33.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 -- 172.21.15.64:0/2092908857 shutdown_connections 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baad94f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad882b0 0x560baad92630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 --2- 172.21.15.64:0/2092908857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad87990 0x560baad87d70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 -- 172.21.15.64:0/2092908857 >> 172.21.15.64:0/2092908857 conn(0x560baad83f20 msgr2=0x560baad84300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 -- 172.21.15.64:0/2092908857 shutdown_connections 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.646+0000 7feb201af740 1 -- 172.21.15.64:0/2092908857 wait complete. 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 -- start start 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 Processor -- start 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad87990 0x560baad8fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baad8ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad92b70 0x560baad90490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560baad95740 con 0x560baad87990 2024-08-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560baac8b7c0 con 0x560baad882b0 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x560baad958c0 con 0x560baad92b70 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb201af740 10 client.0 _ll_register_callbacks cb 0x560baac94f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad87990 0x560baad8fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad87990 0x560baad8fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37336/0 (socket says 172.21.15.64:37336) 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb177fe640 1 -- 172.21.15.64:0/2043347352 learned_addr learned my addr 172.21.15.64:0/2043347352 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb0effd640 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baad8ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.647+0000 7feb17fff640 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad92b70 0x560baad90490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb177fe640 1 -- 172.21.15.64:0/2043347352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad92b70 msgr2=0x560baad90490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb177fe640 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad92b70 0x560baad90490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb177fe640 1 -- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 msgr2=0x560baad8ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb177fe640 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baad8ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb177fe640 1 -- 172.21.15.64:0/2043347352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560baaac7960 con 0x560baad87990 2024-08-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb177fe640 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad87990 0x560baad8fa10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7feb080cc590 tx=0x7feb08005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c19d88fb9875e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feaf6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feaf6ffd640 1 -- 172.21.15.64:0/2043347352 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0809b300 con 0x560baad87990 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feaf6ffd640 1 -- 172.21.15.64:0/2043347352 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9554+0+0 (secure 0 0 0) 0x7feb08097970 con 0x560baad87990 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb10043070 con 0x560baad87990 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feaf6ffd640 1 -- 172.21.15.64:0/2043347352 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb0809f6b0 con 0x560baad87990 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb201af740 10 client.16230 fetch_fsmap learned FSMap version 32 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560baada1790 con 0x560baad87990 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.648+0000 7feaf6ffd640 1 -- 172.21.15.64:0/2043347352 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb10043070 con 0x560baad87990 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feaf6ffd640 1 -- 172.21.15.64:0/2043347352 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7feb080bb110 con 0x560baad87990 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb201af740 10 client.16230 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb201af740 10 client.16230 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb201af740 20 client.16230 populate_metadata read hostname 'smithi064' 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb201af740 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baada2c90 0x560baada5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb201af740 4 client.16230 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560baacfb1d0 con 0x560baada2c90 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb0effd640 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baada2c90 0x560baada5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feb0effd640 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baada2c90 0x560baada5080 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c3004c238524ce2 server_cookie=67dc5372c4656960 in_seq=0 out_seq=0 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.649+0000 7feaf6ffd640 10 client.16230 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feaf6ffd640 1 -- 172.21.15.64:0/2043347352 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560baacfb1d0 con 0x560baada2c90 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feaf6ffd640 10 client.16230 handle_command_reply: tid=0 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 client.16230 shutdown 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad87990 msgr2=0x560baad8fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad87990 0x560baad8fa10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7feb080cc590 tx=0x7feb08005dc0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 shutdown_connections 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad92b70 0x560baad90490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baada2c90 0x560baada5080 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baad8ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 --2- 172.21.15.64:0/2043347352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad87990 0x560baad8fa10 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feaf6ffd640 0 client.16230 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 >> 172.21.15.64:0/2043347352 conn(0x560baad83f20 msgr2=0x560baad89c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 shutdown_connections 2024-08-19T01:50:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 1 -- 172.21.15.64:0/2043347352 wait complete. 2024-08-19T01:50:33.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.650+0000 7feb201af740 20 client.16230 trim_cache size 0 max 16384 2024-08-19T01:50:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.667+0000 7feb201af740 1 Processor -- start 2024-08-19T01:50:33.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.667+0000 7feb201af740 1 -- start start 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb201af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baad918e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb201af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad882b0 0x560baad8e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb201af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad92b70 0x560baadaa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb201af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb181b16a0 con 0x560baad882b0 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb201af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb181aef70 con 0x560baad92b70 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb201af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb181acde0 con 0x560baad87990 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad882b0 0x560baad8e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad882b0 0x560baad8e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37346/0 (socket says 172.21.15.64:37346) 2024-08-19T01:50:33.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 -- 172.21.15.64:0/3716641453 learned_addr learned my addr 172.21.15.64:0/3716641453 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb17fff640 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad92b70 0x560baadaa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 -- 172.21.15.64:0/3716641453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 msgr2=0x560baad918e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baad918e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 -- 172.21.15.64:0/3716641453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad92b70 msgr2=0x560baadaa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad92b70 0x560baadaa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 -- 172.21.15.64:0/3716641453 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560baad8a800 con 0x560baad882b0 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0effd640 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad882b0 0x560baad8e2e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7feb0000cbb0 tx=0x7feb000108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2b5bc13e7f079b server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0d7fa640 1 -- 172.21.15.64:0/3716641453 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00005220 con 0x560baad882b0 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0d7fa640 1 -- 172.21.15.64:0/3716641453 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb00004c80 con 0x560baad882b0 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.668+0000 7feb0d7fa640 1 -- 172.21.15.64:0/3716641453 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0001c8f0 con 0x560baad882b0 2024-08-19T01:50:33.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 -- 172.21.15.64:0/3716641453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad882b0 msgr2=0x560baad8e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad882b0 0x560baad8e2e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7feb0000cbb0 tx=0x7feb000108c0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 -- 172.21.15.64:0/3716641453 shutdown_connections 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad92b70 0x560baadaa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad882b0 0x560baad8e2e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 --2- 172.21.15.64:0/3716641453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baad918e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 -- 172.21.15.64:0/3716641453 >> 172.21.15.64:0/3716641453 conn(0x560baad83f20 msgr2=0x560baad83080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 -- 172.21.15.64:0/3716641453 shutdown_connections 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 -- 172.21.15.64:0/3716641453 wait complete. 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.669+0000 7feb201af740 1 -- start start 2024-08-19T01:50:33.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 1 Processor -- start 2024-08-19T01:50:33.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baae08ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baae072f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baae07830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb181ad430 con 0x560baad92b70 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb181ae8a0 con 0x560baad882b0 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb181adb00 con 0x560baad87990 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb201af740 10 client.0 _ll_register_callbacks cb 0x7feb08097660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb17fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baae07830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb17fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baae07830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37362/0 (socket says 172.21.15.64:37362) 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb177fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baae08ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb17fff640 1 -- 172.21.15.64:0/570850330 learned_addr learned my addr 172.21.15.64:0/570850330 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.670+0000 7feb0effd640 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baae072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb177fe640 1 -- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 msgr2=0x560baae072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb177fe640 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baae072f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb177fe640 1 -- 172.21.15.64:0/570850330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 msgr2=0x560baae07830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:50:33.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb177fe640 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baae07830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb177fe640 1 -- 172.21.15.64:0/570850330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560baae07df0 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb177fe640 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baae08ea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb08097760 tx=0x7feb080cc9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=353d1d0c29884360 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feaf6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feaf6ffd640 1 -- 172.21.15.64:0/570850330 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0809ac60 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feaf6ffd640 1 -- 172.21.15.64:0/570850330 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9554+0+0 (secure 0 0 0) 0x7feb080cc110 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb00004c80 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feaf6ffd640 1 -- 172.21.15.64:0/570850330 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb0809b300 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb201af740 10 client.25630 fetch_fsmap learned FSMap version 32 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560baad9a940 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.671+0000 7feaf6ffd640 1 -- 172.21.15.64:0/570850330 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb00004c80 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feaf6ffd640 1 -- 172.21.15.64:0/570850330 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7feb080b4080 con 0x560baad87990 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb201af740 10 client.25630 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb201af740 10 client.25630 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:50:33.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb201af740 20 client.25630 populate_metadata read hostname 'smithi064' 2024-08-19T01:50:33.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb201af740 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baad9af90 0x560baad9b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:50:33.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb201af740 4 client.25630 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7ef4964d", "release": true, "await": true}] 2024-08-19T01:50:33.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7ef4964d", "release": true, "await": true}) -- 0x560baacf56e0 con 0x560baad9af90 2024-08-19T01:50:33.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb0effd640 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baad9af90 0x560baad9b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:50:33.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feaf6ffd640 10 client.25630 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:33.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.672+0000 7feb0effd640 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baad9af90 0x560baad9b3a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9206f931478abe24 server_cookie=68f2a2017f0b4cd4 in_seq=0 out_seq=0 2024-08-19T01:50:33.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feaf6ffd640 1 -- 172.21.15.64:0/570850330 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560baacf56e0 con 0x560baad9af90 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feaf6ffd640 10 client.25630 handle_command_reply: tid=0 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 client.25630 shutdown 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 msgr2=0x560baae08ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baae08ea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb08097760 tx=0x7feb080cc9e0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 shutdown_connections 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560baad92b70 0x560baae07830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x560baad9af90 0x560baad9b3a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560baad882b0 0x560baae072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 --2- 172.21.15.64:0/570850330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560baad87990 0x560baae08ea0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feaf6ffd640 0 client.25630 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:50:33.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 >> 172.21.15.64:0/570850330 conn(0x560baad83f20 msgr2=0x560baad893e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 shutdown_connections 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.729+0000 7feb201af740 1 -- 172.21.15.64:0/570850330 wait complete. 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.730+0000 7feb201af740 20 client.25630 trim_cache size 0 max 16384 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 32, 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 64, 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-08-19T01:50:33.733 INFO:teuthology.orchestra.run.smithi064.stdout: "7ef4964d": { 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 64, 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 90.0, 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 147.1, 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-08-19T01:50:33.734 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-08-19T01:50:33.735 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-08-19T01:50:33.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:33.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:33.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:33.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:33.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:50:33.735 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.730+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feb040a2cf0 msgr2=0x7feb040a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.730+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feb040a2cf0 0x7feb040a51b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feb1004c000 tx=0x7feb1004d000 comp rx=0 tx=0).stop 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.730+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb181018b0 msgr2=0x7feb18198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.730+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb181018b0 0x7feb18198130 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7feb0800d950 tx=0x7feb0800de10 comp rx=0 tx=0).stop 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 shutdown_connections 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7feb040a2cf0 0x7feb040a51b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb181078b0 0x7feb1819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb18102250 0x7feb18198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 --2- 172.21.15.64:0/942834451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb181018b0 0x7feb18198130 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 >> 172.21.15.64:0/942834451 conn(0x7feb180fd5f0 msgr2=0x7feb180fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:50:33.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 shutdown_connections 2024-08-19T01:50:33.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:50:33.731+0000 7feb1ddb1640 1 -- 172.21.15.64:0/942834451 wait complete. 2024-08-19T01:50:33.737 INFO:tasks.workunit.client.0.smithi064.stdout: 10 65648 15.10 MB/sec execute 125 sec latency 26135.438 ms 2024-08-19T01:50:34.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:34 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:50:34.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:34 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:50:34.367 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:34 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:50:34.395 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7ef4964d', total seconds elapsed: 30.3 2024-08-19T01:50:34.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:34 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:50:34.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:34 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:50:34.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:34 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:50:34.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:34 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:50:34.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:34 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:50:34.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:34 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:50:34.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 66265 15.24 MB/sec execute 126 sec latency 26143.328 ms 2024-08-19T01:50:35.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 66753 15.35 MB/sec execute 127 sec latency 55.728 ms 2024-08-19T01:50:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:35 smithi123 ceph-mon[21829]: pgmap v545: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 237 KiB/s wr, 12 op/s 2024-08-19T01:50:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:35 smithi161 ceph-mon[21757]: pgmap v545: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 237 KiB/s wr, 12 op/s 2024-08-19T01:50:36.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:35 smithi064 ceph-mon[18515]: pgmap v545: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 237 KiB/s wr, 12 op/s 2024-08-19T01:50:36.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 67123 15.39 MB/sec execute 128 sec latency 135.657 ms 2024-08-19T01:50:37.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 67629 15.49 MB/sec execute 129 sec latency 138.334 ms 2024-08-19T01:50:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:37 smithi123 ceph-mon[21829]: pgmap v546: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-08-19T01:50:38.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:37 smithi161 ceph-mon[21757]: pgmap v546: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-08-19T01:50:38.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:37 smithi064 ceph-mon[18515]: pgmap v546: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 70 op/s 2024-08-19T01:50:38.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 68045 15.53 MB/sec execute 130 sec latency 190.521 ms 2024-08-19T01:50:39.733 INFO:tasks.workunit.client.0.smithi064.stdout: 10 68551 15.62 MB/sec execute 131 sec latency 193.638 ms 2024-08-19T01:50:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:39 smithi123 ceph-mon[21829]: pgmap v547: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 890 KiB/s wr, 64 op/s 2024-08-19T01:50:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:39 smithi161 ceph-mon[21757]: pgmap v547: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 890 KiB/s wr, 64 op/s 2024-08-19T01:50:40.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:39 smithi064 ceph-mon[18515]: pgmap v547: 129 pgs: 129 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 890 KiB/s wr, 64 op/s 2024-08-19T01:50:40.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 68972 15.66 MB/sec execute 132 sec latency 316.904 ms 2024-08-19T01:50:41.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 69484 15.74 MB/sec execute 133 sec latency 1316.947 ms 2024-08-19T01:50:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:41 smithi123 ceph-mon[21829]: pgmap v548: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-08-19T01:50:42.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:41 smithi161 ceph-mon[21757]: pgmap v548: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-08-19T01:50:42.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:41 smithi064 ceph-mon[18515]: pgmap v548: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-08-19T01:50:42.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 69956 15.84 MB/sec execute 134 sec latency 2316.985 ms 2024-08-19T01:50:43.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 70399 15.92 MB/sec execute 135 sec latency 3317.013 ms 2024-08-19T01:50:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:43 smithi123 ceph-mon[21829]: pgmap v549: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 237 op/s 2024-08-19T01:50:44.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:43 smithi161 ceph-mon[21757]: pgmap v549: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 237 op/s 2024-08-19T01:50:44.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:43 smithi064 ceph-mon[18515]: pgmap v549: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 237 op/s 2024-08-19T01:50:44.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 70817 15.96 MB/sec execute 136 sec latency 4317.040 ms 2024-08-19T01:50:45.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 71203 15.98 MB/sec execute 137 sec latency 5317.073 ms 2024-08-19T01:50:46.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:45 smithi123 ceph-mon[21829]: pgmap v550: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 236 op/s 2024-08-19T01:50:46.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:45 smithi161 ceph-mon[21757]: pgmap v550: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 236 op/s 2024-08-19T01:50:46.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:45 smithi064 ceph-mon[18515]: pgmap v550: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 236 op/s 2024-08-19T01:50:46.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 71639 16.01 MB/sec execute 138 sec latency 6317.108 ms 2024-08-19T01:50:47.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 72042 16.05 MB/sec execute 139 sec latency 6521.736 ms 2024-08-19T01:50:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:47 smithi123 ceph-mon[21829]: pgmap v551: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 424 op/s 2024-08-19T01:50:48.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:47 smithi161 ceph-mon[21757]: pgmap v551: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 424 op/s 2024-08-19T01:50:48.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:47 smithi064 ceph-mon[18515]: pgmap v551: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 424 op/s 2024-08-19T01:50:48.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 72525 16.14 MB/sec execute 140 sec latency 6304.294 ms 2024-08-19T01:50:49.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:48 smithi123 ceph-mon[21829]: pgmap v552: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 366 op/s 2024-08-19T01:50:49.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:48 smithi161 ceph-mon[21757]: pgmap v552: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 366 op/s 2024-08-19T01:50:49.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:48 smithi064 ceph-mon[18515]: pgmap v552: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 366 op/s 2024-08-19T01:50:49.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 73017 16.18 MB/sec execute 141 sec latency 7304.331 ms 2024-08-19T01:50:50.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 73587 16.32 MB/sec execute 142 sec latency 8304.382 ms 2024-08-19T01:50:51.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:51 smithi123 ceph-mon[21829]: pgmap v553: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.2 MiB/s wr, 433 op/s 2024-08-19T01:50:51.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:51 smithi161 ceph-mon[21757]: pgmap v553: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.2 MiB/s wr, 433 op/s 2024-08-19T01:50:51.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:51 smithi064 ceph-mon[18515]: pgmap v553: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.2 MiB/s wr, 433 op/s 2024-08-19T01:50:51.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 73957 16.39 MB/sec execute 143 sec latency 9304.427 ms 2024-08-19T01:50:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 74316 16.36 MB/sec execute 144 sec latency 9492.244 ms 2024-08-19T01:50:53.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:53 smithi123 ceph-mon[21829]: pgmap v554: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.2 MiB/s wr, 430 op/s 2024-08-19T01:50:53.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:53 smithi161 ceph-mon[21757]: pgmap v554: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.2 MiB/s wr, 430 op/s 2024-08-19T01:50:53.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:53 smithi064 ceph-mon[18515]: pgmap v554: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.2 MiB/s wr, 430 op/s 2024-08-19T01:50:53.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 74755 16.43 MB/sec execute 145 sec latency 187.062 ms 2024-08-19T01:50:54.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 75177 16.37 MB/sec execute 146 sec latency 173.113 ms 2024-08-19T01:50:55.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:55 smithi123 ceph-mon[21829]: pgmap v555: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.1 MiB/s wr, 329 op/s 2024-08-19T01:50:55.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:55 smithi161 ceph-mon[21757]: pgmap v555: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.1 MiB/s wr, 329 op/s 2024-08-19T01:50:55.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:55 smithi064 ceph-mon[18515]: pgmap v555: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.1 MiB/s wr, 329 op/s 2024-08-19T01:50:55.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 75819 16.55 MB/sec execute 147 sec latency 173.229 ms 2024-08-19T01:50:56.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 76305 16.65 MB/sec execute 148 sec latency 130.186 ms 2024-08-19T01:50:57.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:57 smithi123 ceph-mon[21829]: pgmap v556: 129 pgs: 129 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.5 MiB/s wr, 481 op/s 2024-08-19T01:50:57.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:57 smithi161 ceph-mon[21757]: pgmap v556: 129 pgs: 129 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.5 MiB/s wr, 481 op/s 2024-08-19T01:50:57.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:57 smithi064 ceph-mon[18515]: pgmap v556: 129 pgs: 129 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.5 MiB/s wr, 481 op/s 2024-08-19T01:50:57.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 76770 16.70 MB/sec execute 149 sec latency 105.341 ms 2024-08-19T01:50:58.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 77256 16.73 MB/sec execute 150 sec latency 144.950 ms 2024-08-19T01:50:59.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:50:59 smithi123 ceph-mon[21829]: pgmap v557: 129 pgs: 129 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 4.6 MiB/s wr, 293 op/s 2024-08-19T01:50:59.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:50:59 smithi161 ceph-mon[21757]: pgmap v557: 129 pgs: 129 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 4.6 MiB/s wr, 293 op/s 2024-08-19T01:50:59.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:50:59 smithi064 ceph-mon[18515]: pgmap v557: 129 pgs: 129 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 4.6 MiB/s wr, 293 op/s 2024-08-19T01:50:59.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 77770 16.84 MB/sec execute 151 sec latency 161.773 ms 2024-08-19T01:51:00.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 78179 16.87 MB/sec execute 152 sec latency 315.130 ms 2024-08-19T01:51:01.415 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:01 smithi123 ceph-mon[21829]: pgmap v558: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 367 op/s 2024-08-19T01:51:01.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:01 smithi161 ceph-mon[21757]: pgmap v558: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 367 op/s 2024-08-19T01:51:01.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:01 smithi064 ceph-mon[18515]: pgmap v558: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 367 op/s 2024-08-19T01:51:01.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 78179 16.76 MB/sec execute 153 sec latency 1315.167 ms 2024-08-19T01:51:02.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 78188 16.65 MB/sec execute 154 sec latency 2315.273 ms 2024-08-19T01:51:03.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:03 smithi161 ceph-mon[21757]: pgmap v559: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 369 op/s 2024-08-19T01:51:03.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:03 smithi064 ceph-mon[18515]: pgmap v559: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 369 op/s 2024-08-19T01:51:03.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:03 smithi123 ceph-mon[21829]: pgmap v559: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 369 op/s 2024-08-19T01:51:03.734 INFO:tasks.workunit.client.0.smithi064.stdout: 10 78188 16.54 MB/sec execute 155 sec latency 3315.324 ms 2024-08-19T01:51:04.735 INFO:tasks.workunit.client.0.smithi064.stdout: 10 78188 16.44 MB/sec execute 156 sec latency 4315.398 ms 2024-08-19T01:51:05.091 INFO:tasks.workunit.client.0.smithi064.stdout:[68687] write failed on handle 11841 (Resource temporarily unavailable) 2024-08-19T01:51:05.091 INFO:tasks.workunit.client.0.smithi064.stdout:Child failed with status 1 2024-08-19T01:51:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T01:51:05.104 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-08-19T01:51:05.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-19T01:51:05.417 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:05 smithi064 ceph-mon[18515]: pgmap v560: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.6 MiB/s wr, 295 op/s 2024-08-19T01:51:05.419 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi064 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-08-19T01:51:05.610 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e821b3b17ae146a7ac5cba89cd8065ea Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi064 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-08-19T01:51:05.612 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-08-19T01:51:05.624 INFO:tasks.quiescer:joining Quiescers 2024-08-19T01:51:05.624 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-08-19T01:51:05.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:51:05.630 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:05 smithi161 ceph-mon[21757]: pgmap v560: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.6 MiB/s wr, 295 op/s 2024-08-19T01:51:05.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:05 smithi123 ceph-mon[21829]: pgmap v560: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.6 MiB/s wr, 295 op/s 2024-08-19T01:51:05.886 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:06.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 -- 172.21.15.64:0/391734633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 msgr2=0x7f83f4100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:06.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/391734633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4100590 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f83e400aa10 tx=0x7f83e402b720 comp rx=0 tx=0).stop 2024-08-19T01:51:06.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 -- 172.21.15.64:0/391734633 shutdown_connections 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/391734633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83f41052e0 0x7f83f410ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/391734633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83f4100b60 0x7f83f4104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/391734633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4100590 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 -- 172.21.15.64:0/391734633 >> 172.21.15.64:0/391734633 conn(0x7f83f4076f40 msgr2=0x7f83f40fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 -- 172.21.15.64:0/391734633 shutdown_connections 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 -- 172.21.15.64:0/391734633 wait complete. 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.509+0000 7f83fa0b6640 1 Processor -- start 2024-08-19T01:51:06.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83fa0b6640 1 -- start start 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83fa0b6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83fa0b6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83f4100b60 0x7f83f41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83fa0b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83f41052e0 0x7f83f419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83fa0b6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f83f410fa80 con 0x7f83f41001b0 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83fa0b6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f83f410f900 con 0x7f83f4100b60 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83fa0b6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83f410fc00 con 0x7f83f41052e0 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:06.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34232/0 (socket says 172.21.15.64:34232) 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f37fe640 1 -- 172.21.15.64:0/3346303066 learned_addr learned my addr 172.21.15.64:0/3346303066 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f2ffd640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83f4100b60 0x7f83f41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f3fff640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83f41052e0 0x7f83f419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f37fe640 1 -- 172.21.15.64:0/3346303066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83f41052e0 msgr2=0x7f83f419ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f37fe640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83f41052e0 0x7f83f419ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f37fe640 1 -- 172.21.15.64:0/3346303066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83f4100b60 msgr2=0x7f83f41986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.510+0000 7f83f37fe640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83f4100b60 0x7f83f41986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.511+0000 7f83f37fe640 1 -- 172.21.15.64:0/3346303066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f419d140 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.511+0000 7f83f37fe640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4198190 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f83e4006010 tx=0x7f83e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.511+0000 7f83f0ff9640 1 -- 172.21.15.64:0/3346303066 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e403f070 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.511+0000 7f83f0ff9640 1 -- 172.21.15.64:0/3346303066 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e4032400 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.511+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f419d3d0 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.511+0000 7f83f0ff9640 1 -- 172.21.15.64:0/3346303066 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4036040 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.511+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f41a4d10 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.512+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f4101690 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.513+0000 7f83f0ff9640 1 -- 172.21.15.64:0/3346303066 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f83e4042070 con 0x7f83f41001b0 2024-08-19T01:51:06.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.514+0000 7f83f0ff9640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f83dc0a2cf0 0x7f83dc0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:06.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.514+0000 7f83f2ffd640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f83dc0a2cf0 0x7f83dc0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:06.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.514+0000 7f83f0ff9640 1 -- 172.21.15.64:0/3346303066 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9554+0+0 (secure 0 0 0) 0x7f83e40f0060 con 0x7f83f41001b0 2024-08-19T01:51:06.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.514+0000 7f83f2ffd640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f83dc0a2cf0 0x7f83dc0a51b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f83e8005fd0 tx=0x7f83e8005850 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:06.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.518+0000 7f83f0ff9640 1 -- 172.21.15.64:0/3346303066 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f83e40bc870 con 0x7f83f41001b0 2024-08-19T01:51:06.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.851+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83f41084c0 con 0x7f83f41001b0 2024-08-19T01:51:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.857+0000 7f83f0ff9640 1 -- 172.21.15.64:0/3346303066 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11921 (secure 0 0 0) 0x7f83e40bc170 con 0x7f83f41001b0 2024-08-19T01:51:06.858 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:51:06.858 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":32,"btime":"2024-08-19T01:49:07:146032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:49:06.572445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:51:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 32 2024-08-19T01:51:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.862+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f83dc0a2cf0 msgr2=0x7f83dc0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.862+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f83dc0a2cf0 0x7f83dc0a51b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f83e8005fd0 tx=0x7f83e8005850 comp rx=0 tx=0).stop 2024-08-19T01:51:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.862+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 msgr2=0x7f83f4198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.862+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4198190 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f83e4006010 tx=0x7f83e40042f0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.865+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 shutdown_connections 2024-08-19T01:51:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.865+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f83dc0a2cf0 0x7f83dc0a51b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.865+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83f41052e0 0x7f83f419ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.865+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83f4100b60 0x7f83f41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.865+0000 7f83fa0b6640 1 --2- 172.21.15.64:0/3346303066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f41001b0 0x7f83f4198190 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.865+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 >> 172.21.15.64:0/3346303066 conn(0x7f83f4076f40 msgr2=0x7f83f4103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.866+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 shutdown_connections 2024-08-19T01:51:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:06.870+0000 7f83fa0b6640 1 -- 172.21.15.64:0/3346303066 wait complete. 2024-08-19T01:51:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:07 smithi161 ceph-mon[21757]: pgmap v561: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 540 op/s 2024-08-19T01:51:07.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:07 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3346303066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:07.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:07 smithi064 ceph-mon[18515]: pgmap v561: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 540 op/s 2024-08-19T01:51:07.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:07 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3346303066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:07.578 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24476 quiesce db --cancel --all' 2024-08-19T01:51:07.578 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- timeout 105 ceph tell mds.24476 quiesce db --cancel --all 2024-08-19T01:51:07.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:07 smithi123 ceph-mon[21829]: pgmap v561: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 540 op/s 2024-08-19T01:51:07.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:07 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3346303066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:07.928 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:08.369 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:08 smithi064 ceph-mon[18515]: osdmap e117: 12 total, 12 up, 12 in 2024-08-19T01:51:08.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:08 smithi161 ceph-mon[21757]: osdmap e117: 12 total, 12 up, 12 in 2024-08-19T01:51:08.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/239127762 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0103410 msgr2=0x7fe4a01078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/239127762 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0103410 0x7fe4a01078c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe488009930 tx=0x7fe48802a420 comp rx=0 tx=0).stop 2024-08-19T01:51:08.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/239127762 shutdown_connections 2024-08-19T01:51:08.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/239127762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0107fd0 0x7fe4a010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/239127762 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0103410 0x7fe4a01078c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/239127762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0102a60 0x7fe4a0102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/239127762 >> 172.21.15.64:0/239127762 conn(0x7fe4a00fc500 msgr2=0x7fe4a00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/239127762 shutdown_connections 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.457+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/239127762 wait complete. 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.458+0000 7fe4a4c2f640 1 Processor -- start 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.458+0000 7fe4a4c2f640 1 -- start start 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe4a4c2f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0102a60 0x7fe4a01982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe4a4c2f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0103410 0x7fe4a0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe4a4c2f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0107fd0 0x7fe4a019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0102a60 0x7fe4a01982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0102a60 0x7fe4a01982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:48702/0 (socket says 172.21.15.64:48702) 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0103410 0x7fe4a0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0107fd0 0x7fe4a019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0103410 0x7fe4a0198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.64:35894/0 (socket says 172.21.15.64:35894) 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0107fd0 0x7fe4a019cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34250/0 (socket says 172.21.15.64:34250) 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49dd9b640 1 -- 172.21.15.64:0/1594451562 learned_addr learned my addr 172.21.15.64:0/1594451562 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a0111dd0 con 0x7fe4a0107fd0 2024-08-19T01:51:08.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a0111c50 con 0x7fe4a0103410 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a0111f50 con 0x7fe4a0102a60 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49e59c640 1 -- 172.21.15.64:0/1594451562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0103410 msgr2=0x7fe4a0198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49e59c640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0103410 0x7fe4a0198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49e59c640 1 -- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0107fd0 msgr2=0x7fe4a019cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49e59c640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0107fd0 0x7fe4a019cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49e59c640 1 -- 172.21.15.64:0/1594451562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a019d2a0 con 0x7fe4a0102a60 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0107fd0 0x7fe4a019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:51:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.459+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0103410 0x7fe4a0198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.460+0000 7fe49e59c640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0102a60 0x7fe4a01982f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe49000eb00 tx=0x7fe49000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.460+0000 7fe4977fe640 1 -- 172.21.15.64:0/1594451562 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490010430 con 0x7fe4a0102a60 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.460+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a019d590 con 0x7fe4a0102a60 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.460+0000 7fe4977fe640 1 -- 172.21.15.64:0/1594451562 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe490010ab0 con 0x7fe4a0102a60 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.460+0000 7fe4977fe640 1 -- 172.21.15.64:0/1594451562 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490017bb0 con 0x7fe4a0102a60 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.460+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a01a4e70 con 0x7fe4a0102a60 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.463+0000 7fe4a702d740 1 Processor -- start 2024-08-19T01:51:08.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.463+0000 7fe4a702d740 1 -- start start 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a2743116a0 0x55a274311a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a274311fc0 0x55a27431c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27431c880 0x55a27431ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4977fe640 1 -- 172.21.15.64:0/1594451562 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe490010c60 con 0x7fe4a0102a60 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4a702d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a2742728c0 con 0x55a274311fc0 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4a702d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55a2742249e0 con 0x55a27431c880 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4a702d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a274280540 con 0x55a2743116a0 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a274311fc0 0x55a27431c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a2743116a0 0x55a274311a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a274311fc0 0x55a27431c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34266/0 (socket says 172.21.15.64:34266) 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a2743116a0 0x55a274311a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:48714/0 (socket says 172.21.15.64:48714) 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 -- 172.21.15.64:0/885491252 learned_addr learned my addr 172.21.15.64:0/885491252 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27431c880 0x55a27431ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 -- 172.21.15.64:0/885491252 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a2743116a0 msgr2=0x55a274311a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a2743116a0 0x55a274311a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 -- 172.21.15.64:0/885491252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27431c880 msgr2=0x55a27431ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27431c880 0x55a27431ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 -- 172.21.15.64:0/885491252 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27420e640 con 0x55a274311fc0 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe4977fe640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe48c0a2cf0 0x7fe48c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.464+0000 7fe49e59c640 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a274311fc0 0x55a27431c340 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe4900cd4c0 tx=0x7fe4900d87c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f651a45afd8ddd server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe48c0a2cf0 0x7fe48c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe494ff9640 1 -- 172.21.15.64:0/885491252 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4900985c0 con 0x55a274311fc0 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4977fe640 1 -- 172.21.15.64:0/1594451562 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fe4900d0260 con 0x7fe4a0102a60 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe494ff9640 1 -- 172.21.15.64:0/885491252 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe49009ca20 con 0x55a274311fc0 2024-08-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe494ff9640 1 -- 172.21.15.64:0/885491252 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49009fe10 con 0x55a274311fc0 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4a702d740 1 -- 172.21.15.64:0/885491252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a274311fc0 msgr2=0x55a27431c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe48c0a2cf0 0x7fe48c0a51b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe498012560 tx=0x7fe498005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4a702d740 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a274311fc0 0x55a27431c340 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe4900cd4c0 tx=0x7fe4900d87c0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4a702d740 1 -- 172.21.15.64:0/885491252 shutdown_connections 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4a702d740 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27431c880 0x55a27431ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4a702d740 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a274311fc0 0x55a27431c340 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4a702d740 1 --2- 172.21.15.64:0/885491252 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a2743116a0 0x55a274311a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.465+0000 7fe4a702d740 1 -- 172.21.15.64:0/885491252 >> 172.21.15.64:0/885491252 conn(0x55a27430dc30 msgr2=0x55a27430e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 -- 172.21.15.64:0/885491252 shutdown_connections 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 -- 172.21.15.64:0/885491252 wait complete. 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 -- start start 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 Processor -- start 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a27431c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a2743193b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27431c880 0x55a2743198f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a274275c10 con 0x55a27431c880 2024-08-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55a274234f00 con 0x55a2743116a0 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a27427af30 con 0x55a274311fc0 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe4a702d740 10 client.0 _ll_register_callbacks cb 0x55a273f85b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe49ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27431c880 0x55a2743198f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe49ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27431c880 0x55a2743198f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34276/0 (socket says 172.21.15.64:34276) 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.466+0000 7fe49ed9d640 1 -- 172.21.15.64:0/2174838382 learned_addr learned my addr 172.21.15.64:0/2174838382 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49e59c640 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a27431c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a2743193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49ed9d640 1 -- 172.21.15.64:0/2174838382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 msgr2=0x55a2743193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a2743193b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49ed9d640 1 -- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 msgr2=0x55a27431c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a27431c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49ed9d640 1 -- 172.21.15.64:0/2174838382 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a273fe8620 con 0x55a27431c880 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27431c880 0x55a2743198f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe498003670 tx=0x7fe498004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f34351dfc477f65 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe485ffb640 1 -- 172.21.15.64:0/2174838382 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49800f850 con 0x55a27431c880 2024-08-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe49009ca20 con 0x55a27431c880 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe485ffb640 1 -- 172.21.15.64:0/2174838382 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fe49800f9f0 con 0x55a27431c880 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe485ffb640 1 -- 172.21.15.64:0/2174838382 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe49800fbf0 con 0x55a27431c880 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe4a702d740 10 client.16254 fetch_fsmap learned FSMap version 32 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a27432f290 con 0x55a27431c880 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.467+0000 7fe485ffb640 1 -- 172.21.15.64:0/2174838382 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe49009ca20 con 0x55a27431c880 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe485ffb640 1 -- 172.21.15.64:0/2174838382 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe498027070 con 0x55a27431c880 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe4a702d740 10 client.16254 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:51:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe4a702d740 10 client.16254 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe4a702d740 20 client.16254 populate_metadata read hostname 'smithi064' 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a2743306a0 0x55a274332ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe4a702d740 4 client.16254 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a274283e70 con 0x55a2743306a0 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe49e59c640 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a2743306a0 0x55a274332ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe485ffb640 10 client.16254 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.468+0000 7fe49e59c640 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a2743306a0 0x55a274332ab0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d7bf5701493a6b server_cookie=1aae97828dfff03e in_seq=0 out_seq=0 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe485ffb640 1 -- 172.21.15.64:0/2174838382 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a274283e70 con 0x55a2743306a0 2024-08-19T01:51:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe485ffb640 10 client.16254 handle_command_reply: tid=0 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 client.16254 shutdown 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27431c880 msgr2=0x55a2743198f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27431c880 0x55a2743198f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe498003670 tx=0x7fe498004730 comp rx=0 tx=0).stop 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 shutdown_connections 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27431c880 0x55a2743198f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a2743193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a2743306a0 0x55a274332ab0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2174838382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a27431c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe485ffb640 0 client.16254 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:51:08.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 >> 172.21.15.64:0/2174838382 conn(0x55a27430dc30 msgr2=0x55a274313910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:08.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 shutdown_connections 2024-08-19T01:51:08.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 1 -- 172.21.15.64:0/2174838382 wait complete. 2024-08-19T01:51:08.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.469+0000 7fe4a702d740 20 client.16254 trim_cache size 0 max 16384 2024-08-19T01:51:08.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 Processor -- start 2024-08-19T01:51:08.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 -- start start 2024-08-19T01:51:08.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27430dc30 0x55a274318c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a274338350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274338890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a01ad070 con 0x55a27430dc30 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a01aaee0 con 0x55a2743116a0 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe4a702d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a01ac9a0 con 0x55a274311fc0 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27430dc30 0x55a274318c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27430dc30 0x55a274318c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34284/0 (socket says 172.21.15.64:34284) 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 -- 172.21.15.64:0/3907567313 learned_addr learned my addr 172.21.15.64:0/3907567313 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274338890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a274338350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 -- 172.21.15.64:0/3907567313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 msgr2=0x55a274338890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274338890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 -- 172.21.15.64:0/3907567313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 msgr2=0x55a274338350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a274338350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 -- 172.21.15.64:0/3907567313 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a274333100 con 0x55a27430dc30 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a274338350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.487+0000 7fe49e59c640 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27430dc30 0x55a274318c90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe49000c7d0 tx=0x7fe490005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a6d57d577a4c35 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe487fff640 1 -- 172.21.15.64:0/3907567313 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4900a0470 con 0x55a27430dc30 2024-08-19T01:51:08.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe487fff640 1 -- 172.21.15.64:0/3907567313 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4900d3b80 con 0x55a27430dc30 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe487fff640 1 -- 172.21.15.64:0/3907567313 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4900a06d0 con 0x55a27430dc30 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 -- 172.21.15.64:0/3907567313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27430dc30 msgr2=0x55a274318c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27430dc30 0x55a274318c90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe49000c7d0 tx=0x7fe490005f10 comp rx=0 tx=0).stop 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 -- 172.21.15.64:0/3907567313 shutdown_connections 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274338890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a2743116a0 0x55a274338350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 --2- 172.21.15.64:0/3907567313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a27430dc30 0x55a274318c90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 -- 172.21.15.64:0/3907567313 >> 172.21.15.64:0/3907567313 conn(0x55a274319a50 msgr2=0x55a274314230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 -- 172.21.15.64:0/3907567313 shutdown_connections 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.488+0000 7fe4a702d740 1 -- 172.21.15.64:0/3907567313 wait complete. 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 -- start start 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 Processor -- start 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27430dc30 0x55a2743248e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2743116a0 0x55a274324e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274325360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a01abc00 con 0x55a2743116a0 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a01ab530 con 0x55a27430dc30 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a01ac2d0 con 0x55a274311fc0 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe4a702d740 10 client.0 _ll_register_callbacks cb 0x7fe498009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2743116a0 0x55a274324e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2743116a0 0x55a274324e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34296/0 (socket says 172.21.15.64:34296) 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 -- 172.21.15.64:0/2585710795 learned_addr learned my addr 172.21.15.64:0/2585710795 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49e59c640 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27430dc30 0x55a2743248e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49ed9d640 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274325360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 -- 172.21.15.64:0/2585710795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 msgr2=0x55a274325360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274325360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 -- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27430dc30 msgr2=0x55a2743248e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27430dc30 0x55a2743248e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.489+0000 7fe49dd9b640 1 -- 172.21.15.64:0/2585710795 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27439a570 con 0x55a2743116a0 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe49dd9b640 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2743116a0 0x55a274324e20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe488009230 tx=0x7fe488031980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b68b5e301581a62 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe485ffb640 1 -- 172.21.15.64:0/2585710795 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48804b020 con 0x55a2743116a0 2024-08-19T01:51:08.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe485ffb640 1 -- 172.21.15.64:0/2585710795 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fe488004450 con 0x55a2743116a0 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4900d3b80 con 0x55a2743116a0 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe485ffb640 1 -- 172.21.15.64:0/2585710795 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe488053050 con 0x55a2743116a0 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe4a702d740 10 client.16260 fetch_fsmap learned FSMap version 32 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a27439b870 con 0x55a2743116a0 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe485ffb640 1 -- 172.21.15.64:0/2585710795 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4900d3b80 con 0x55a2743116a0 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe485ffb640 1 -- 172.21.15.64:0/2585710795 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe488046070 con 0x55a2743116a0 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe4a702d740 10 client.16260 fetch_fsmap finished waiting for FSMap version 32 2024-08-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe4a702d740 10 client.16260 resolve_mds: validated gid 24476 aka daemon mds.b 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.490+0000 7fe4a702d740 20 client.16260 populate_metadata read hostname 'smithi064' 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a27431c880 0x55a27439e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe4a702d740 4 client.16260 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 --> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55a27427e380 con 0x55a27431c880 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe49e59c640 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a27431c880 0x55a27439e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe49e59c640 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a27431c880 0x55a27439e0e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a48e6083ec8de67e server_cookie=2e98ad5098d2fddd in_seq=0 out_seq=0 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe485ffb640 10 client.16260 ms_handle_connect on v2:172.21.15.123:6838/2281975354 2024-08-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe485ffb640 1 -- 172.21.15.64:0/2585710795 <== mds.0 v2:172.21.15.123:6838/2281975354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55a27427e380 con 0x55a27431c880 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe485ffb640 10 client.16260 handle_command_reply: tid=0 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe4a702d740 1 client.16260 shutdown 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2743116a0 msgr2=0x55a274324e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.491+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2743116a0 0x55a274324e20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe488009230 tx=0x7fe488031980 comp rx=0 tx=0).stop 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 shutdown_connections 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a274311fc0 0x55a274325360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a2743116a0 0x55a274324e20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:6838/2281975354,v1:172.21.15.123:6839/2281975354] conn(0x55a27431c880 0x55a27439e0e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe485ffb640 0 client.16260 ms_handle_reset on v2:172.21.15.123:6838/2281975354 2024-08-19T01:51:08.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 --2- 172.21.15.64:0/2585710795 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a27430dc30 0x55a2743248e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 >> 172.21.15.64:0/2585710795 conn(0x55a274319a50 msgr2=0x55a274313060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:08.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 shutdown_connections 2024-08-19T01:51:08.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 1 -- 172.21.15.64:0/2585710795 wait complete. 2024-08-19T01:51:08.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a702d740 20 client.16260 trim_cache size 0 max 16384 2024-08-19T01:51:08.516 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:51:08.517 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 32, 2024-08-19T01:51:08.517 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24476, 2024-08-19T01:51:08.517 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 64, 2024-08-19T01:51:08.517 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": {} 2024-08-19T01:51:08.517 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:51:08.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe48c0a2cf0 msgr2=0x7fe48c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe48c0a2cf0 0x7fe48c0a51b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe498012560 tx=0x7fe498005f90 comp rx=0 tx=0).stop 2024-08-19T01:51:08.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0102a60 msgr2=0x7fe4a01982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:08.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0102a60 0x7fe4a01982f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe49000eb00 tx=0x7fe49000c2d0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 shutdown_connections 2024-08-19T01:51:08.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fe48c0a2cf0 0x7fe48c0a51b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4a0107fd0 0x7fe4a019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0103410 0x7fe4a0198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 --2- 172.21.15.64:0/1594451562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a0102a60 0x7fe4a01982f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:08.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 >> 172.21.15.64:0/1594451562 conn(0x7fe4a00fc500 msgr2=0x7fe4a0105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:08.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 shutdown_connections 2024-08-19T01:51:08.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:08.492+0000 7fe4a4c2f640 1 -- 172.21.15.64:0/1594451562 wait complete. 2024-08-19T01:51:08.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:08 smithi123 ceph-mon[21829]: osdmap e117: 12 total, 12 up, 12 in 2024-08-19T01:51:09.224 INFO:tasks.quiescer:done joining Quiescers 2024-08-19T01:51:09.224 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-08-19T01:51:09.234 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-08-19T01:51:09.234 INFO:teuthology.task.exec:Executing custom commands... 2024-08-19T01:51:09.234 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:51:09.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-08-19T01:51:09.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-08-19T01:51:09.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-08-19T01:51:09.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 -- 172.21.15.64:0/1924690643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c0105050 msgr2=0x7f04c010cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 --2- 172.21.15.64:0/1924690643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c0105050 0x7f04c010cba0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009ca0 tx=0x7f04b802a8c0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 -- 172.21.15.64:0/1924690643 shutdown_connections 2024-08-19T01:51:09.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 --2- 172.21.15.64:0/1924690643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c0105050 0x7f04c010cba0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 --2- 172.21.15.64:0/1924690643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c01008b0 0x7f04c01048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 --2- 172.21.15.64:0/1924690643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c00fff00 0x7f04c01002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 -- 172.21.15.64:0/1924690643 >> 172.21.15.64:0/1924690643 conn(0x7f04c0076d20 msgr2=0x7f04c00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:09.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 -- 172.21.15.64:0/1924690643 shutdown_connections 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.477+0000 7f04c759d640 1 -- 172.21.15.64:0/1924690643 wait complete. 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 Processor -- start 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 -- start start 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c00fff00 0x7f04c0198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c0105050 0x7f04c0199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c0199a00 0x7f04c01a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01101c0 con 0x7f04c00fff00 2024-08-19T01:51:09.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04c0110040 con 0x7f04c0105050 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c759d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04c0110340 con 0x7f04c0199a00 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c659b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c00fff00 0x7f04c0198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c659b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c00fff00 0x7f04c0198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41718/0 (socket says 172.21.15.64:41718) 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c659b640 1 -- 172.21.15.64:0/3323355950 learned_addr learned my addr 172.21.15.64:0/3323355950 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.478+0000 7f04c5d9a640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c0105050 0x7f04c0199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c6d9c640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c0199a00 0x7f04c01a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c659b640 1 -- 172.21.15.64:0/3323355950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c0199a00 msgr2=0x7f04c01a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c659b640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c0199a00 0x7f04c01a3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c659b640 1 -- 172.21.15.64:0/3323355950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c0105050 msgr2=0x7f04c0199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c659b640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c0105050 0x7f04c0199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c659b640 1 -- 172.21.15.64:0/3323355950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c01a4780 con 0x7f04c00fff00 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c659b640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c00fff00 0x7f04c0198f30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00b4a0 tx=0x7f04bc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04b77fe640 1 -- 172.21.15.64:0/3323355950 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc015020 con 0x7f04c00fff00 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04b77fe640 1 -- 172.21.15.64:0/3323355950 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04bc011d10 con 0x7f04c00fff00 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c01a4a70 con 0x7f04c00fff00 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04b77fe640 1 -- 172.21.15.64:0/3323355950 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00f2e0 con 0x7f04c00fff00 2024-08-19T01:51:09.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.479+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c01a5010 con 0x7f04c00fff00 2024-08-19T01:51:09.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.480+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c0100360 con 0x7f04c00fff00 2024-08-19T01:51:09.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.481+0000 7f04b77fe640 1 -- 172.21.15.64:0/3323355950 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f04bc018020 con 0x7f04c00fff00 2024-08-19T01:51:09.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.482+0000 7f04b77fe640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04ac0a2d70 0x7f04ac0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.482+0000 7f04c5d9a640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04ac0a2d70 0x7f04ac0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.482+0000 7f04b77fe640 1 -- 172.21.15.64:0/3323355950 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f04bc0ceff0 con 0x7f04c00fff00 2024-08-19T01:51:09.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.482+0000 7f04c5d9a640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04ac0a2d70 0x7f04ac0a5230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f04c0103a70 tx=0x7f04b0009110 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:09.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:09 smithi064 ceph-mon[18515]: pgmap v563: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 12 MiB/s wr, 465 op/s 2024-08-19T01:51:09.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.486+0000 7f04b77fe640 1 -- 172.21.15.64:0/3323355950 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f04bc01f020 con 0x7f04c00fff00 2024-08-19T01:51:09.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:09 smithi123 ceph-mon[21829]: pgmap v563: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 12 MiB/s wr, 465 op/s 2024-08-19T01:51:09.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.680+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f04c0100920 con 0x7f04ac0a2d70 2024-08-19T01:51:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.690+0000 7f04b77fe640 1 -- 172.21.15.64:0/3323355950 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f04c0100920 con 0x7f04ac0a2d70 2024-08-19T01:51:09.691 INFO:teuthology.orchestra.run.smithi064.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-08-19T00:00:00", "created": "2024-08-19T01:33:16", "first": "2024-08-19T01:34:00", "last": "2024-08-19T01:51:00", "last_pruned": "2024-08-19T01:51:02", "created_count": 18, "pruned_count": 12, "active": true} 2024-08-19T01:51:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04ac0a2d70 msgr2=0x7f04ac0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04ac0a2d70 0x7f04ac0a5230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f04c0103a70 tx=0x7f04b0009110 comp rx=0 tx=0).stop 2024-08-19T01:51:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c00fff00 msgr2=0x7f04c0198f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c00fff00 0x7f04c0198f30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00b4a0 tx=0x7f04bc00b8a0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 shutdown_connections 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f04ac0a2d70 0x7f04ac0a5230 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c0199a00 0x7f04c01a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c0105050 0x7f04c0199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 --2- 172.21.15.64:0/3323355950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04c00fff00 0x7f04c0198f30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 >> 172.21.15.64:0/3323355950 conn(0x7f04c0076d20 msgr2=0x7f04c00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 shutdown_connections 2024-08-19T01:51:09.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.692+0000 7f04c759d640 1 -- 172.21.15.64:0/3323355950 wait complete. 2024-08-19T01:51:09.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:09 smithi161 ceph-mon[21757]: pgmap v563: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 12 MiB/s wr, 465 op/s 2024-08-19T01:51:09.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-08-19T01:51:09.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.805+0000 7f02d78a4640 1 -- 172.21.15.64:0/104569669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d01066f0 msgr2=0x7f02d0106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 --2- 172.21.15.64:0/104569669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d01066f0 0x7f02d0106ad0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009930 tx=0x7f02c402a420 comp rx=0 tx=0).stop 2024-08-19T01:51:09.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 -- 172.21.15.64:0/104569669 shutdown_connections 2024-08-19T01:51:09.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 --2- 172.21.15.64:0/104569669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02d0101080 0x7f02d010d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 --2- 172.21.15.64:0/104569669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02d01006e0 0x7f02d0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 --2- 172.21.15.64:0/104569669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d01066f0 0x7f02d0106ad0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 -- 172.21.15.64:0/104569669 >> 172.21.15.64:0/104569669 conn(0x7f02d00776c0 msgr2=0x7f02d00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:09.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 -- 172.21.15.64:0/104569669 shutdown_connections 2024-08-19T01:51:09.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 -- 172.21.15.64:0/104569669 wait complete. 2024-08-19T01:51:09.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 Processor -- start 2024-08-19T01:51:09.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 -- start start 2024-08-19T01:51:09.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02d01006e0 0x7f02d0197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.806+0000 7f02d78a4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d0101080 0x7f02d01975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d78a4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02d01066f0 0x7f02d019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d78a4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f02d0110b80 con 0x7f02d0101080 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d78a4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02d0110a00 con 0x7f02d01066f0 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d78a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f02d0110d00 con 0x7f02d01006e0 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d0101080 0x7f02d01975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d0101080 0x7f02d01975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41730/0 (socket says 172.21.15.64:41730) 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 -- 172.21.15.64:0/4134684141 learned_addr learned my addr 172.21.15.64:0/4134684141 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:09.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d5640640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02d01006e0 0x7f02d0197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d5e41640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02d01066f0 0x7f02d019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 -- 172.21.15.64:0/4134684141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02d01006e0 msgr2=0x7f02d0197060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02d01006e0 0x7f02d0197060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 -- 172.21.15.64:0/4134684141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02d01066f0 msgr2=0x7f02d019b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:09.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02d01066f0 0x7f02d019b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:09.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 -- 172.21.15.64:0/4134684141 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d019c050 con 0x7f02d0101080 2024-08-19T01:51:09.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d5e41640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02d01066f0 0x7f02d019b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:51:09.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d4e3f640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d0101080 0x7f02d01975a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00b4a0 tx=0x7f02cc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:09.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02c27fc640 1 -- 172.21.15.64:0/4134684141 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc014070 con 0x7f02d0101080 2024-08-19T01:51:09.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02c27fc640 1 -- 172.21.15.64:0/4134684141 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02cc010af0 con 0x7f02d0101080 2024-08-19T01:51:09.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02c27fc640 1 -- 172.21.15.64:0/4134684141 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc017d10 con 0x7f02d0101080 2024-08-19T01:51:09.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d019c340 con 0x7f02d0101080 2024-08-19T01:51:09.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.807+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d01a3bd0 con 0x7f02d0101080 2024-08-19T01:51:09.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.808+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02d0100b40 con 0x7f02d0101080 2024-08-19T01:51:09.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.810+0000 7f02c27fc640 1 -- 172.21.15.64:0/4134684141 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f02cc017070 con 0x7f02d0101080 2024-08-19T01:51:09.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.810+0000 7f02c27fc640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02bc0a2da0 0x7f02bc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:09.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.810+0000 7f02d5640640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02bc0a2da0 0x7f02bc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:09.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.810+0000 7f02c27fc640 1 -- 172.21.15.64:0/4134684141 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f02cc0ceb70 con 0x7f02d0101080 2024-08-19T01:51:09.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.811+0000 7f02d5640640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02bc0a2da0 0x7f02bc0a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02c4006010 tx=0x7f02c4000bc0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:09.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:09.814+0000 7f02c27fc640 1 -- 172.21.15.64:0/4134684141 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f02cc09b440 con 0x7f02d0101080 2024-08-19T01:51:10.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.014+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 --> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f02d0108cb0 con 0x7f02bc0a2da0 2024-08-19T01:51:10.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.024+0000 7f02c27fc640 1 -- 172.21.15.64:0/4134684141 <== mgr.15258 v2:172.21.15.64:6800/3917059587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f02d0108cb0 con 0x7f02bc0a2da0 2024-08-19T01:51:10.025 INFO:teuthology.orchestra.run.smithi064.stdout:/ 1m 6m3h 2024-08-19T01:51:10.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.025+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02bc0a2da0 msgr2=0x7f02bc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02bc0a2da0 0x7f02bc0a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02c4006010 tx=0x7f02c4000bc0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d0101080 msgr2=0x7f02d01975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d0101080 0x7f02d01975a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00b4a0 tx=0x7f02cc00b960 comp rx=0 tx=0).stop 2024-08-19T01:51:10.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 shutdown_connections 2024-08-19T01:51:10.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f02bc0a2da0 0x7f02bc0a5260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02d01066f0 0x7f02d019b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02d0101080 0x7f02d01975a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 --2- 172.21.15.64:0/4134684141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02d01006e0 0x7f02d0197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 >> 172.21.15.64:0/4134684141 conn(0x7f02d00776c0 msgr2=0x7f02d01035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:10.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 shutdown_connections 2024-08-19T01:51:10.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.026+0000 7f02d78a4640 1 -- 172.21.15.64:0/4134684141 wait complete. 2024-08-19T01:51:10.037 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-08-19T01:51:10.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T01:51:10.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 -- 172.21.15.64:0/1248382861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c1065f0 msgr2=0x7fa25c1069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 --2- 172.21.15.64:0/1248382861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c1065f0 0x7fa25c1069d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009900 tx=0x7fa24c02a410 comp rx=0 tx=0).stop 2024-08-19T01:51:10.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 -- 172.21.15.64:0/1248382861 shutdown_connections 2024-08-19T01:51:10.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 --2- 172.21.15.64:0/1248382861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa25c100ed0 0x7fa25c10d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 --2- 172.21.15.64:0/1248382861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa25c100530 0x7fa25c100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 --2- 172.21.15.64:0/1248382861 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c1065f0 0x7fa25c1069d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 -- 172.21.15.64:0/1248382861 >> 172.21.15.64:0/1248382861 conn(0x7fa25c0fc250 msgr2=0x7fa25c0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:10.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 -- 172.21.15.64:0/1248382861 shutdown_connections 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.148+0000 7fa25bfff640 1 -- 172.21.15.64:0/1248382861 wait complete. 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 Processor -- start 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 -- start start 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c100530 0x7fa25c198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa25c100ed0 0x7fa25c199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa25c1065f0 0x7fa25c1a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c110990 con 0x7fa25c100530 2024-08-19T01:51:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c110810 con 0x7fa25c1065f0 2024-08-19T01:51:10.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25bfff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c110b10 con 0x7fa25c100ed0 2024-08-19T01:51:10.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c100530 0x7fa25c198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c100530 0x7fa25c198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41742/0 (socket says 172.21.15.64:41742) 2024-08-19T01:51:10.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.149+0000 7fa25affd640 1 -- 172.21.15.64:0/2195019439 learned_addr learned my addr 172.21.15.64:0/2195019439 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:10.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25affd640 1 -- 172.21.15.64:0/2195019439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa25c100ed0 msgr2=0x7fa25c199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25a7fc640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa25c100ed0 0x7fa25c199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25affd640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa25c100ed0 0x7fa25c199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25affd640 1 -- 172.21.15.64:0/2195019439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa25c1065f0 msgr2=0x7fa25c1a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25affd640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa25c1065f0 0x7fa25c1a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25affd640 1 -- 172.21.15.64:0/2195019439 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25c1a4280 con 0x7fa25c100530 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25affd640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c100530 0x7fa25c198cc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa24c02aa00 tx=0x7fa24c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa2609fd640 1 -- 172.21.15.64:0/2195019439 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c03f070 con 0x7fa25c100530 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa2609fd640 1 -- 172.21.15.64:0/2195019439 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa24c031c30 con 0x7fa25c100530 2024-08-19T01:51:10.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa2609fd640 1 -- 172.21.15.64:0/2195019439 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c035030 con 0x7fa25c100530 2024-08-19T01:51:10.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25c1a4450 con 0x7fa25c100530 2024-08-19T01:51:10.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.150+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa25c1a4910 con 0x7fa25c100530 2024-08-19T01:51:10.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.151+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa25c100990 con 0x7fa25c100530 2024-08-19T01:51:10.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.153+0000 7fa2609fd640 1 -- 172.21.15.64:0/2195019439 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa24c042070 con 0x7fa25c100530 2024-08-19T01:51:10.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.153+0000 7fa2609fd640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa2440a2d80 0x7fa2440a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.154+0000 7fa25a7fc640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa2440a2d80 0x7fa2440a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.154+0000 7fa2609fd640 1 -- 172.21.15.64:0/2195019439 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fa24c0f06b0 con 0x7fa25c100530 2024-08-19T01:51:10.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.154+0000 7fa25a7fc640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa2440a2d80 0x7fa2440a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa25c06a9c0 tx=0x7fa250005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:10.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.157+0000 7fa2609fd640 1 -- 172.21.15.64:0/2195019439 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fa24c031de0 con 0x7fa25c100530 2024-08-19T01:51:10.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.335+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa25c1a4c70 con 0x7fa25c100530 2024-08-19T01:51:10.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.337+0000 7fa2609fd640 1 -- 172.21.15.64:0/2195019439 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+25533 (secure 0 0 0) 0x7fa24c0bce10 con 0x7fa25c100530 2024-08-19T01:51:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:51:10.338 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":117,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:51:07.185820+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":117,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":117,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/2663741075":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1810737759":"2024-08-20T01:33:13.815234+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1532949280":"2024-08-20T01:33:13.815234+0000","172.21.15.64:6800/2663741075":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1697666313":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1603406100":"2024-08-20T01:33:13.815234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]},{"pool":4,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]},{"pool":4,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:51:10.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.338+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa2440a2d80 msgr2=0x7fa2440a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.338+0000 7fa25bfff640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa2440a2d80 0x7fa2440a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa25c06a9c0 tx=0x7fa250005f90 comp rx=0 tx=0).stop 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.338+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c100530 msgr2=0x7fa25c198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.338+0000 7fa25bfff640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c100530 0x7fa25c198cc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa24c02aa00 tx=0x7fa24c003c10 comp rx=0 tx=0).stop 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 shutdown_connections 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fa2440a2d80 0x7fa2440a5240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa25c1065f0 0x7fa25c1a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa25c100ed0 0x7fa25c199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 --2- 172.21.15.64:0/2195019439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25c100530 0x7fa25c198cc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 >> 172.21.15.64:0/2195019439 conn(0x7fa25c0fc250 msgr2=0x7fa25c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:10.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 shutdown_connections 2024-08-19T01:51:10.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.339+0000 7fa25bfff640 1 -- 172.21.15.64:0/2195019439 wait complete. 2024-08-19T01:51:10.349 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:10 smithi064 ceph-mon[18515]: from='client.16266 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:51:10.351 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T01:26:37.913262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-08-19T01:30:47.899062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-08-19T01:30:50.489331+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 31, 'snap_epoch': 117, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-08-19T01:30:56.822547+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 31, 'snap_epoch': 117, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-08-19T01:51:10.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-19T01:51:10.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.455+0000 7fee93d54640 1 -- 172.21.15.64:0/2594028876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c100ec0 msgr2=0x7fee8c10db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.455+0000 7fee93d54640 1 --2- 172.21.15.64:0/2594028876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c100ec0 0x7fee8c10db80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009900 tx=0x7fee7c02a410 comp rx=0 tx=0).stop 2024-08-19T01:51:10.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.455+0000 7fee93d54640 1 -- 172.21.15.64:0/2594028876 shutdown_connections 2024-08-19T01:51:10.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.455+0000 7fee93d54640 1 --2- 172.21.15.64:0/2594028876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee8c10e0c0 0x7fee8c1104b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.455+0000 7fee93d54640 1 --2- 172.21.15.64:0/2594028876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c100ec0 0x7fee8c10db80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.455+0000 7fee93d54640 1 --2- 172.21.15.64:0/2594028876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c1005a0 0x7fee8c100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.455+0000 7fee93d54640 1 -- 172.21.15.64:0/2594028876 >> 172.21.15.64:0/2594028876 conn(0x7fee8c0fc400 msgr2=0x7fee8c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 -- 172.21.15.64:0/2594028876 shutdown_connections 2024-08-19T01:51:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 -- 172.21.15.64:0/2594028876 wait complete. 2024-08-19T01:51:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 Processor -- start 2024-08-19T01:51:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 -- start start 2024-08-19T01:51:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c1005a0 0x7fee8c19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c100ec0 0x7fee8c19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee8c10e0c0 0x7fee8c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c113e70 con 0x7fee8c1005a0 2024-08-19T01:51:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c113cf0 con 0x7fee8c100ec0 2024-08-19T01:51:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee93d54640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c113ff0 con 0x7fee8c10e0c0 2024-08-19T01:51:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee91af0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c1005a0 0x7fee8c19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee91af0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c1005a0 0x7fee8c19a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41756/0 (socket says 172.21.15.64:41756) 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.456+0000 7fee91af0640 1 -- 172.21.15.64:0/4227063858 learned_addr learned my addr 172.21.15.64:0/4227063858 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee912ef640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c100ec0 0x7fee8c19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee91af0640 1 -- 172.21.15.64:0/4227063858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee8c10e0c0 msgr2=0x7fee8c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee91af0640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee8c10e0c0 0x7fee8c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee91af0640 1 -- 172.21.15.64:0/4227063858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c100ec0 msgr2=0x7fee8c19a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee91af0640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c100ec0 0x7fee8c19a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee91af0640 1 -- 172.21.15.64:0/4227063858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8c19f3b0 con 0x7fee8c1005a0 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee912ef640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c100ec0 0x7fee8c19a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee91af0640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c1005a0 0x7fee8c19a3c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fee8800d950 tx=0x7fee8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee82ffd640 1 -- 172.21.15.64:0/4227063858 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88017070 con 0x7fee8c1005a0 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee82ffd640 1 -- 172.21.15.64:0/4227063858 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee88009c30 con 0x7fee8c1005a0 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee8c19f6a0 con 0x7fee8c1005a0 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee82ffd640 1 -- 172.21.15.64:0/4227063858 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8800f6b0 con 0x7fee8c1005a0 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.457+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee8c1a6f30 con 0x7fee8c1005a0 2024-08-19T01:51:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.458+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee8c100980 con 0x7fee8c1005a0 2024-08-19T01:51:10.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.459+0000 7fee82ffd640 1 -- 172.21.15.64:0/4227063858 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fee880109d0 con 0x7fee8c1005a0 2024-08-19T01:51:10.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.460+0000 7fee82ffd640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fee780a2d00 0x7fee780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.460+0000 7fee912ef640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fee780a2d00 0x7fee780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.460+0000 7fee82ffd640 1 -- 172.21.15.64:0/4227063858 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fee880cf5a0 con 0x7fee8c1005a0 2024-08-19T01:51:10.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.461+0000 7fee912ef640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fee780a2d00 0x7fee780a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fee7c000c00 tx=0x7fee7c005a20 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:10.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.463+0000 7fee82ffd640 1 -- 172.21.15.64:0/4227063858 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fee88009de0 con 0x7fee8c1005a0 2024-08-19T01:51:10.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.639+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee8c02d5e0 con 0x7fee8c1005a0 2024-08-19T01:51:10.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.640+0000 7fee82ffd640 1 -- 172.21.15.64:0/4227063858 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7fee88097030 con 0x7fee8c1005a0 2024-08-19T01:51:10.641 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-08-19T01:51:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fee780a2d00 msgr2=0x7fee780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fee780a2d00 0x7fee780a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fee7c000c00 tx=0x7fee7c005a20 comp rx=0 tx=0).stop 2024-08-19T01:51:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c1005a0 msgr2=0x7fee8c19a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c1005a0 0x7fee8c19a3c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fee8800d950 tx=0x7fee8800de10 comp rx=0 tx=0).stop 2024-08-19T01:51:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 shutdown_connections 2024-08-19T01:51:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fee780a2d00 0x7fee780a51c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee8c10e0c0 0x7fee8c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c100ec0 0x7fee8c19a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 --2- 172.21.15.64:0/4227063858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee8c1005a0 0x7fee8c19a3c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 >> 172.21.15.64:0/4227063858 conn(0x7fee8c0fc400 msgr2=0x7fee8c0fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:10.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 shutdown_connections 2024-08-19T01:51:10.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.642+0000 7fee93d54640 1 -- 172.21.15.64:0/4227063858 wait complete. 2024-08-19T01:51:10.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-08-19T01:51:10.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:10 smithi123 ceph-mon[21829]: from='client.16266 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:51:10.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:10 smithi161 ceph-mon[21757]: from='client.16266 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:51:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.756+0000 7fd99e5c6640 1 -- 172.21.15.64:0/519614070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9981017b0 msgr2=0x7fd998101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.756+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/519614070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9981017b0 0x7fd998101c10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009930 tx=0x7fd98c02a420 comp rx=0 tx=0).stop 2024-08-19T01:51:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.756+0000 7fd99e5c6640 1 -- 172.21.15.64:0/519614070 shutdown_connections 2024-08-19T01:51:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.756+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/519614070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd998102150 0x7fd99810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.757+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/519614070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9981017b0 0x7fd998101c10 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.757+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/519614070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9981077c0 0x7fd998107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.757+0000 7fd99e5c6640 1 -- 172.21.15.64:0/519614070 >> 172.21.15.64:0/519614070 conn(0x7fd9980fd530 msgr2=0x7fd9980ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.757+0000 7fd99e5c6640 1 -- 172.21.15.64:0/519614070 shutdown_connections 2024-08-19T01:51:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.757+0000 7fd99e5c6640 1 -- 172.21.15.64:0/519614070 wait complete. 2024-08-19T01:51:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.757+0000 7fd99e5c6640 1 Processor -- start 2024-08-19T01:51:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.757+0000 7fd99e5c6640 1 -- start start 2024-08-19T01:51:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd99e5c6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9981017b0 0x7fd998198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd99e5c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd998102150 0x7fd998198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd99e5c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9981077c0 0x7fd99819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd99e5c6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd998111c00 con 0x7fd998102150 2024-08-19T01:51:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd99e5c6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd998111a80 con 0x7fd9981017b0 2024-08-19T01:51:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd99e5c6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd998111d80 con 0x7fd9981077c0 2024-08-19T01:51:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd998102150 0x7fd998198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd998102150 0x7fd998198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41764/0 (socket says 172.21.15.64:41764) 2024-08-19T01:51:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 -- 172.21.15.64:0/4201666349 learned_addr learned my addr 172.21.15.64:0/4201666349 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd99cb63640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9981077c0 0x7fd99819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd997fff640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9981017b0 0x7fd998198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 -- 172.21.15.64:0/4201666349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9981077c0 msgr2=0x7fd99819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9981077c0 0x7fd99819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 -- 172.21.15.64:0/4201666349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9981017b0 msgr2=0x7fd998198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9981017b0 0x7fd998198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 -- 172.21.15.64:0/4201666349 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99819d120 con 0x7fd998102150 2024-08-19T01:51:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9977fe640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd998102150 0x7fd998198670 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009230 tx=0x7fd98c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9957fa640 1 -- 172.21.15.64:0/4201666349 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c03f030 con 0x7fd998102150 2024-08-19T01:51:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9957fa640 1 -- 172.21.15.64:0/4201666349 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd98c003ad0 con 0x7fd998102150 2024-08-19T01:51:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.758+0000 7fd9957fa640 1 -- 172.21.15.64:0/4201666349 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c035070 con 0x7fd998102150 2024-08-19T01:51:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.759+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99819d3b0 con 0x7fd998102150 2024-08-19T01:51:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.759+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9981a4ca0 con 0x7fd998102150 2024-08-19T01:51:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.759+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd998102ef0 con 0x7fd998102150 2024-08-19T01:51:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.761+0000 7fd9957fa640 1 -- 172.21.15.64:0/4201666349 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd98c042030 con 0x7fd998102150 2024-08-19T01:51:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.762+0000 7fd9957fa640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd9800a2d00 0x7fd9800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.762+0000 7fd997fff640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd9800a2d00 0x7fd9800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.762+0000 7fd9957fa640 1 -- 172.21.15.64:0/4201666349 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7fd98c0f0100 con 0x7fd998102150 2024-08-19T01:51:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.762+0000 7fd997fff640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd9800a2d00 0x7fd9800a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd988005fd0 tx=0x7fd98800b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:10.765+0000 7fd9957fa640 1 -- 172.21.15.64:0/4201666349 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fd98c0bc270 con 0x7fd998102150 2024-08-19T01:51:11.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.029+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd998109d20 con 0x7fd998102150 2024-08-19T01:51:11.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.029+0000 7fd9957fa640 1 -- 172.21.15.64:0/4201666349 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) ==== 104+0+11 (secure 0 0 0) 0x7fd98c0c0570 con 0x7fd998102150 2024-08-19T01:51:11.031 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-08-19T01:51:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd9800a2d00 msgr2=0x7fd9800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd9800a2d00 0x7fd9800a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd988005fd0 tx=0x7fd98800b390 comp rx=0 tx=0).stop 2024-08-19T01:51:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd998102150 msgr2=0x7fd998198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd998102150 0x7fd998198670 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009230 tx=0x7fd98c004270 comp rx=0 tx=0).stop 2024-08-19T01:51:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 shutdown_connections 2024-08-19T01:51:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fd9800a2d00 0x7fd9800a51c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9981077c0 0x7fd99819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd998102150 0x7fd998198670 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 --2- 172.21.15.64:0/4201666349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9981017b0 0x7fd998198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 >> 172.21.15.64:0/4201666349 conn(0x7fd9980fd530 msgr2=0x7fd9980fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 shutdown_connections 2024-08-19T01:51:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.032+0000 7fd99e5c6640 1 -- 172.21.15.64:0/4201666349 wait complete. 2024-08-19T01:51:11.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-08-19T01:51:11.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 -- 172.21.15.64:0/1767836214 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301006b0 msgr2=0x7ff330100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1767836214 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301006b0 0x7ff330100b10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff324009930 tx=0x7ff32402a420 comp rx=0 tx=0).stop 2024-08-19T01:51:11.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 -- 172.21.15.64:0/1767836214 shutdown_connections 2024-08-19T01:51:11.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1767836214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff330101050 0x7ff33010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1767836214 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301006b0 0x7ff330100b10 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1767836214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301066c0 0x7ff330106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 -- 172.21.15.64:0/1767836214 >> 172.21.15.64:0/1767836214 conn(0x7ff330077700 msgr2=0x7ff3300feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:11.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 -- 172.21.15.64:0/1767836214 shutdown_connections 2024-08-19T01:51:11.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 -- 172.21.15.64:0/1767836214 wait complete. 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 Processor -- start 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.146+0000 7ff337b5b640 1 -- start start 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301006b0 0x7ff330199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff330101050 0x7ff330199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301066c0 0x7ff3301a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff330110b50 con 0x7ff3301066c0 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301109d0 con 0x7ff3301006b0 2024-08-19T01:51:11.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff330110cd0 con 0x7ff330101050 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301066c0 0x7ff3301a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301066c0 0x7ff3301a40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41782/0 (socket says 172.21.15.64:41782) 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 -- 172.21.15.64:0/1146510980 learned_addr learned my addr 172.21.15.64:0/1146510980 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3358f7640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301006b0 0x7ff330199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3350f6640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff330101050 0x7ff330199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 -- 172.21.15.64:0/1146510980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff330101050 msgr2=0x7ff330199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff330101050 0x7ff330199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 -- 172.21.15.64:0/1146510980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301006b0 msgr2=0x7ff330199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301006b0 0x7ff330199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 -- 172.21.15.64:0/1146510980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3301a45e0 con 0x7ff3301066c0 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff3360f8640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301066c0 0x7ff3301a40a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff32800c970 tx=0x7ff32800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff32effd640 1 -- 172.21.15.64:0/1146510980 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328007bb0 con 0x7ff3301066c0 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff32effd640 1 -- 172.21.15.64:0/1146510980 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff328007d50 con 0x7ff3301066c0 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3301a47b0 con 0x7ff3301066c0 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff32effd640 1 -- 172.21.15.64:0/1146510980 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32800f450 con 0x7ff3301066c0 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.147+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3301a4cd0 con 0x7ff3301066c0 2024-08-19T01:51:11.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.148+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff330100b10 con 0x7ff3301066c0 2024-08-19T01:51:11.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.150+0000 7ff32effd640 1 -- 172.21.15.64:0/1146510980 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff328004f00 con 0x7ff3301066c0 2024-08-19T01:51:11.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.150+0000 7ff32effd640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3180a2d00 0x7ff3180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.150+0000 7ff3358f7640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3180a2d00 0x7ff3180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.150+0000 7ff32effd640 1 -- 172.21.15.64:0/1146510980 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7ff3280cf920 con 0x7ff3301066c0 2024-08-19T01:51:11.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.151+0000 7ff3358f7640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3180a2d00 0x7ff3180a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff320005fd0 tx=0x7ff320009760 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:11.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.153+0000 7ff32effd640 1 -- 172.21.15.64:0/1146510980 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff328014030 con 0x7ff3301066c0 2024-08-19T01:51:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.330+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff33002d5e0 con 0x7ff3301066c0 2024-08-19T01:51:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.330+0000 7ff32effd640 1 -- 172.21.15.64:0/1146510980 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) ==== 100+0+11 (secure 0 0 0) 0x7ff32809c0e0 con 0x7ff3301066c0 2024-08-19T01:51:11.331 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-08-19T01:51:11.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3180a2d00 msgr2=0x7ff3180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3180a2d00 0x7ff3180a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff320005fd0 tx=0x7ff320009760 comp rx=0 tx=0).stop 2024-08-19T01:51:11.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301066c0 msgr2=0x7ff3301a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301066c0 0x7ff3301a40a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff32800c970 tx=0x7ff32800ce30 comp rx=0 tx=0).stop 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 shutdown_connections 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3180a2d00 0x7ff3180a51c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3301066c0 0x7ff3301a40a0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff330101050 0x7ff330199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 --2- 172.21.15.64:0/1146510980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301006b0 0x7ff330199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 >> 172.21.15.64:0/1146510980 conn(0x7ff330077700 msgr2=0x7ff3301035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:11.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 shutdown_connections 2024-08-19T01:51:11.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.332+0000 7ff337b5b640 1 -- 172.21.15.64:0/1146510980 wait complete. 2024-08-19T01:51:11.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:11 smithi064 ceph-mon[18515]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:51:11.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:11 smithi064 ceph-mon[18515]: pgmap v564: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 429 op/s 2024-08-19T01:51:11.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:11 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2195019439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:51:11.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:11 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4227063858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T01:51:11.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:11 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/4201666349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-08-19T01:51:11.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-08-19T01:51:11.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 -- 172.21.15.64:0/4075171520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e01032a0 msgr2=0x7f97e010b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 --2- 172.21.15.64:0/4075171520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e01032a0 0x7f97e010b780 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009900 tx=0x7f97d402a480 comp rx=0 tx=0).stop 2024-08-19T01:51:11.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 -- 172.21.15.64:0/4075171520 shutdown_connections 2024-08-19T01:51:11.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 --2- 172.21.15.64:0/4075171520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97e010bdd0 0x7f97e010e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 --2- 172.21.15.64:0/4075171520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e01032a0 0x7f97e010b780 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 --2- 172.21.15.64:0/4075171520 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97e0102980 0x7f97e0102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 -- 172.21.15.64:0/4075171520 >> 172.21.15.64:0/4075171520 conn(0x7f97e00fc390 msgr2=0x7f97e00fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 -- 172.21.15.64:0/4075171520 shutdown_connections 2024-08-19T01:51:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.447+0000 7f97e52bc640 1 -- 172.21.15.64:0/4075171520 wait complete. 2024-08-19T01:51:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 Processor -- start 2024-08-19T01:51:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 -- start start 2024-08-19T01:51:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97e0102980 0x7f97e01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97e01032a0 0x7f97e0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e010bdd0 0x7f97e019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0111bb0 con 0x7f97e010bdd0 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0111a30 con 0x7f97e01032a0 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97e52bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0111d30 con 0x7f97e0102980 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e010bdd0 0x7f97e019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e010bdd0 0x7f97e019c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41788/0 (socket says 172.21.15.64:41788) 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97df7fe640 1 -- 172.21.15.64:0/2796795582 learned_addr learned my addr 172.21.15.64:0/2796795582 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97de7fc640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97e01032a0 0x7f97e0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.448+0000 7f97deffd640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97e0102980 0x7f97e01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97df7fe640 1 -- 172.21.15.64:0/2796795582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97e0102980 msgr2=0x7f97e01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97df7fe640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97e0102980 0x7f97e01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97df7fe640 1 -- 172.21.15.64:0/2796795582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97e01032a0 msgr2=0x7f97e0198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97df7fe640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97e01032a0 0x7f97e0198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97df7fe640 1 -- 172.21.15.64:0/2796795582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e019d0d0 con 0x7f97e010bdd0 2024-08-19T01:51:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97df7fe640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e010bdd0 0x7f97e019c9a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f97d800d750 tx=0x7f97d800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97c7fff640 1 -- 172.21.15.64:0/2796795582 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8017070 con 0x7f97e010bdd0 2024-08-19T01:51:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97c7fff640 1 -- 172.21.15.64:0/2796795582 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d8005500 con 0x7f97e010bdd0 2024-08-19T01:51:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97c7fff640 1 -- 172.21.15.64:0/2796795582 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8004d90 con 0x7f97e010bdd0 2024-08-19T01:51:11.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e019d3c0 con 0x7f97e010bdd0 2024-08-19T01:51:11.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.449+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e01a4c50 con 0x7f97e010bdd0 2024-08-19T01:51:11.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.450+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e0102d60 con 0x7f97e010bdd0 2024-08-19T01:51:11.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.451+0000 7f97c7fff640 1 -- 172.21.15.64:0/2796795582 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f97d8004f30 con 0x7f97e010bdd0 2024-08-19T01:51:11.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.452+0000 7f97c7fff640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f97c80a2d00 0x7f97c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.452+0000 7f97deffd640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f97c80a2d00 0x7f97c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.452+0000 7f97c7fff640 1 -- 172.21.15.64:0/2796795582 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f97d80cf170 con 0x7f97e010bdd0 2024-08-19T01:51:11.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.453+0000 7f97deffd640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f97c80a2d00 0x7f97c80a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f97d0005fd0 tx=0x7f97d0009500 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:11.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.456+0000 7f97c7fff640 1 -- 172.21.15.64:0/2796795582 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f97d809b2e0 con 0x7f97e010bdd0 2024-08-19T01:51:11.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.634+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f97e0106e80 con 0x7f97e010bdd0 2024-08-19T01:51:11.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.634+0000 7f97c7fff640 1 -- 172.21.15.64:0/2796795582 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v117) ==== 103+0+11 (secure 0 0 0) 0x7f97d809f5e0 con 0x7f97e010bdd0 2024-08-19T01:51:11.635 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-08-19T01:51:11.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f97c80a2d00 msgr2=0x7f97c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f97c80a2d00 0x7f97c80a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f97d0005fd0 tx=0x7f97d0009500 comp rx=0 tx=0).stop 2024-08-19T01:51:11.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e010bdd0 msgr2=0x7f97e019c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e010bdd0 0x7f97e019c9a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f97d800d750 tx=0x7f97d800dc10 comp rx=0 tx=0).stop 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 shutdown_connections 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f97c80a2d00 0x7f97c80a51c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97e010bdd0 0x7f97e019c9a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97e01032a0 0x7f97e0198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 --2- 172.21.15.64:0/2796795582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97e0102980 0x7f97e01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 >> 172.21.15.64:0/2796795582 conn(0x7f97e00fc390 msgr2=0x7f97e00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:11.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 shutdown_connections 2024-08-19T01:51:11.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.636+0000 7f97e52bc640 1 -- 172.21.15.64:0/2796795582 wait complete. 2024-08-19T01:51:11.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-08-19T01:51:11.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:11 smithi123 ceph-mon[21829]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:51:11.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:11 smithi123 ceph-mon[21829]: pgmap v564: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 429 op/s 2024-08-19T01:51:11.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:11 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2195019439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:51:11.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:11 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4227063858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T01:51:11.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:11 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/4201666349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-08-19T01:51:11.698 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:11 smithi161 ceph-mon[21757]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T01:51:11.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:11 smithi161 ceph-mon[21757]: pgmap v564: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 429 op/s 2024-08-19T01:51:11.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:11 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2195019439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:51:11.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:11 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4227063858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T01:51:11.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:11 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/4201666349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-08-19T01:51:11.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 -- 172.21.15.64:0/425821042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 msgr2=0x7f132c10e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 --2- 172.21.15.64:0/425821042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c10e270 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1324009ca0 tx=0x7f132402a950 comp rx=0 tx=0).stop 2024-08-19T01:51:11.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 -- 172.21.15.64:0/425821042 shutdown_connections 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 --2- 172.21.15.64:0/425821042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c10e270 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 --2- 172.21.15.64:0/425821042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1032b0 0x7f132c10b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 --2- 172.21.15.64:0/425821042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f132c102990 0x7f132c102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 -- 172.21.15.64:0/425821042 >> 172.21.15.64:0/425821042 conn(0x7f132c0fc400 msgr2=0x7f132c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 -- 172.21.15.64:0/425821042 shutdown_connections 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.747+0000 7f133414d640 1 -- 172.21.15.64:0/425821042 wait complete. 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 Processor -- start 2024-08-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 -- start start 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c102990 0x7f132c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f132c1032b0 0x7f132c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f132c111b90 con 0x7f132c10bdb0 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f132c111a10 con 0x7f132c1032b0 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f133414d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f132c111d10 con 0x7f132c102990 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f13326ea640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f13326ea640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c19c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41816/0 (socket says 172.21.15.64:41816) 2024-08-19T01:51:11.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f13326ea640 1 -- 172.21.15.64:0/2323108208 learned_addr learned my addr 172.21.15.64:0/2323108208 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f1331ee9640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c102990 0x7f132c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.748+0000 7f13316e8640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f132c1032b0 0x7f132c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f13326ea640 1 -- 172.21.15.64:0/2323108208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c102990 msgr2=0x7f132c198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f13326ea640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c102990 0x7f132c198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f13326ea640 1 -- 172.21.15.64:0/2323108208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f132c1032b0 msgr2=0x7f132c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f13326ea640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f132c1032b0 0x7f132c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f13326ea640 1 -- 172.21.15.64:0/2323108208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c19d100 con 0x7f132c10bdb0 2024-08-19T01:51:11.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f13326ea640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c19c9d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f132402ae70 tx=0x7f132402f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:11.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f131effd640 1 -- 172.21.15.64:0/2323108208 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324034040 con 0x7f132c10bdb0 2024-08-19T01:51:11.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c19d390 con 0x7f132c10bdb0 2024-08-19T01:51:11.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f131effd640 1 -- 172.21.15.64:0/2323108208 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1324030920 con 0x7f132c10bdb0 2024-08-19T01:51:11.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f131effd640 1 -- 172.21.15.64:0/2323108208 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324045480 con 0x7f132c10bdb0 2024-08-19T01:51:11.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.749+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c1a4c80 con 0x7f132c10bdb0 2024-08-19T01:51:11.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.750+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c102d70 con 0x7f132c10bdb0 2024-08-19T01:51:11.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.751+0000 7f131effd640 1 -- 172.21.15.64:0/2323108208 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1324037020 con 0x7f132c10bdb0 2024-08-19T01:51:11.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.752+0000 7f131effd640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f13180a2d00 0x7f13180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:11.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.752+0000 7f1331ee9640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f13180a2d00 0x7f13180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:11.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.752+0000 7f131effd640 1 -- 172.21.15.64:0/2323108208 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9642+0+0 (secure 0 0 0) 0x7f13240f0770 con 0x7f132c10bdb0 2024-08-19T01:51:11.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.752+0000 7f1331ee9640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f13180a2d00 0x7f13180a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f132000bbe0 tx=0x7f132000b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:11.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.755+0000 7f131effd640 1 -- 172.21.15.64:0/2323108208 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1324080df0 con 0x7f132c10bdb0 2024-08-19T01:51:11.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.977+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f132c106e10 con 0x7f132c10bdb0 2024-08-19T01:51:11.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.985+0000 7f131effd640 1 -- 172.21.15.64:0/2323108208 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740902 (secure 0 0 0) 0x7f13240bd040 con 0x7f132c10bdb0 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 26, 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "active_gid": 15258, 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "active_name": "x", 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "active_addrs": { 2024-08-19T01:51:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3917059587 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-08-19T01:51:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3917059587 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "active_addr": "172.21.15.64:6801/3917059587", 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "active_change": "2024-08-19T01:33:13.815508+0000", 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "active_mgr_features": 4540701547738038271, 2024-08-19T01:51:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "available": true, 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "standbys": [ 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "gid": 24784, 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "z", 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_features": 4540701547738038271, 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "available_modules": [ 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "alerts", 2024-08-19T01:51:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How frequently to reexamine health status", 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_destination": { 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_destination", 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email address to send alerts to", 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_from_name": { 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_from_name", 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "Ceph", 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email From: name", 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_host": { 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_host", 2024-08-19T01:51:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP server", 2024-08-19T01:51:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_password": { 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_password", 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Password to authenticate with", 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_port": { 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_port", 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "465", 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP port", 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_sender": { 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_sender", 2024-08-19T01:51:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP envelope sender", 2024-08-19T01:51:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_ssl": { 2024-08-19T01:51:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_ssl", 2024-08-19T01:51:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use SSL to connect to SMTP server", 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_user": { 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_user", 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "User to authenticate as", 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "balancer", 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "active": { 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "active", 2024-08-19T01:51:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically balance PGs across cluster", 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_time": { 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_time", 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0000", 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "beginning time of day to automatically balance", 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_weekday": { 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_weekday", 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-08-19T01:51:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_max_iterations": { 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_max_iterations", 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "25", 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "250", 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_metrics": { 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_metrics", 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "pgs,objects,bytes", 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_step": { 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_step", 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.5", 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0.001", 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "0.999", 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "aggressiveness of optimization", 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "end_time": { 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_time", 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2359", 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ending time of day to automatically balance", 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-19T01:51:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "end_weekday": { 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_weekday", 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "min_score": { 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_score", 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "upmap", 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-compat", 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "none", 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "read", 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap", 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap-read" 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Balancer mode", 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_ids": { 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_ids", 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_deviation": { 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_deviation", 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "deviation below which no optimization is attempted", 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-08-19T01:51:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_optimizations": { 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_optimizations", 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm", 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_down_multiplier": { 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_down_multiplier", 2024-08-19T01:51:12.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-08-19T01:51:12.058 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_refresh_rate": { 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_refresh_rate", 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.059 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "20", 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_starting_port": { 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_starting_port", 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4721", 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.061 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-08-19T01:51:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_ptrace": { 2024-08-19T01:51:12.062 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_ptrace", 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-08-19T01:51:12.063 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_interval": { 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_interval", 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to autotune daemon memory", 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_memory_target_ratio": { 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_memory_target_ratio", 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.7", 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.066 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_log_destination": { 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm_log_destination", 2024-08-19T01:51:12.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "file", 2024-08-19T01:51:12.068 INFO:teuthology.orchestra.run.smithi064.stdout: "file,syslog", 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: "syslog" 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: "cgroups_split": { 2024-08-19T01:51:12.069 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cgroups_split", 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-08-19T01:51:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: "config_checks_enabled": { 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_checks_enabled", 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.071 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "config_dashboard": { 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_dashboard", 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_alertmanager": { 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_alertmanager", 2024-08-19T01:51:12.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.075 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_base": { 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_base", 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.076 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/ceph", 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Container image name, without the tag", 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_elasticsearch": { 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_elasticsearch", 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.078 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.079 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "elasticsearch container image", 2024-08-19T01:51:12.079 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.079 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.079 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_grafana": { 2024-08-19T01:51:12.079 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_grafana", 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.080 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_haproxy": { 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_haproxy", 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-08-19T01:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "HAproxy container image", 2024-08-19T01:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_agent": { 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_agent", 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-08-19T01:51:12.083 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger agent container image", 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_collector": { 2024-08-19T01:51:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_collector", 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.085 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger collector container image", 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_query": { 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_query", 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger query container image", 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.087 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_keepalived": { 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_keepalived", 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.088 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Keepalived container image", 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_loki": { 2024-08-19T01:51:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_loki", 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Loki container image", 2024-08-19T01:51:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_node_exporter": { 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_node_exporter", 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_nvmeof": { 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_nvmeof", 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.093 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Nvme-of container image", 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_prometheus": { 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_prometheus", 2024-08-19T01:51:12.094 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.095 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_promtail": { 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_promtail", 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.096 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Promtail container image", 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.097 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_samba": { 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_samba", 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.098 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Samba/SMB container image", 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_snmp_gateway": { 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_snmp_gateway", 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.099 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SNMP Gateway container image", 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.100 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "container_init": { 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_init", 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Run podman/docker with `--init`", 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "daemon_cache_timeout": { 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "daemon_cache_timeout", 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.103 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: "default_cephadm_command_timeout": { 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_cephadm_command_timeout", 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.104 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "900", 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.105 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "default_registry": { 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_registry", 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io", 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.106 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: "device_cache_timeout": { 2024-08-19T01:51:12.107 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_cache_timeout", 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1800", 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.108 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache device inventory", 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "device_enhanced_scan": { 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_enhanced_scan", 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use libstoragemgmt during device scans", 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.110 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "facts_cache_timeout": { 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "facts_cache_timeout", 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.111 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache host facts data", 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: "grafana_dashboards_path": { 2024-08-19T01:51:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "grafana_dashboards_path", 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-08-19T01:51:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: "host_check_interval": { 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "host_check_interval", 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to perform a host check", 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "hw_monitoring": { 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hw_monitoring", 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.116 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "inventory_list_all": { 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "inventory_list_all", 2024-08-19T01:51:12.117 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.118 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.118 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.118 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.118 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.118 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.118 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.118 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "log_refresh_metadata": { 2024-08-19T01:51:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_refresh_metadata", 2024-08-19T01:51:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf": { 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "*", 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "max_count_per_host": { 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_count_per_host", 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of daemons per service per host", 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd_draining_count": { 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_osd_draining_count", 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "migration_current": { 2024-08-19T01:51:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "migration_current", 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "internal - do not modify", 2024-08-19T01:51:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "root", 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm-package", 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "root" 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "mode for remote execution of cephadm", 2024-08-19T01:51:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "oob_default_addr": { 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "oob_default_addr", 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "169.254.1.1", 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default address for RedFish API (oob management).", 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "prometheus_alerts_path": { 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus_alerts_path", 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-08-19T01:51:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_insecure": { 2024-08-19T01:51:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_insecure", 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-08-19T01:51:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_password": { 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_password", 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_url": { 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_url", 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_username": { 2024-08-19T01:51:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_username", 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-08-19T01:51:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "secure_monitoring_stack": { 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "secure_monitoring_stack", 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "service_discovery_port": { 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "service_discovery_port", 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8765", 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "cephadm service discovery port", 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ssh_config_file": { 2024-08-19T01:51:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssh_config_file", 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-08-19T01:51:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "use_agent": { 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_agent", 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "use_repo_digest": { 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_repo_digest", 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_failed_host_check": { 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_failed_host_check", 2024-08-19T01:51:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if the host check fails", 2024-08-19T01:51:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_daemons": { 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_daemons", 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_hosts": { 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_hosts", 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crash", 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "retain_interval": { 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retain_interval", 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "31536000", 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain crashes before pruning them", 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_recent_interval": { 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_recent_interval", 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1209600", 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "time interval in which to warn about recent crashes", 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dashboard", 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-08-19T01:51:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_HOST": { 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_HOST", 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-08-19T01:51:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-08-19T01:51:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_ENABLED": { 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_ENABLED", 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-08-19T01:51:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "ENABLE_BROWSABLE_API": { 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ENABLE_BROWSABLE_API", 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-08-19T01:51:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_NFS": { 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-08-19T01:51:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RBD": { 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RGW": { 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_PASSWORD": { 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_PASSWORD", 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-08-19T01:51:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_URL": { 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_URL", 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_USERNAME": { 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_USERNAME", 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-08-19T01:51:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "ISSUE_TRACKER_API_KEY": { 2024-08-19T01:51:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_HOST": { 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_HOST", 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-08-19T01:51:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-08-19T01:51:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_ENABLED": { 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_ENABLED", 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-08-19T01:51:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8", 2024-08-19T01:51:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "REST_REQUESTS_TIMEOUT": { 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "45", 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ACCESS_KEY": { 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ACCESS_KEY", 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SECRET_KEY": { 2024-08-19T01:51:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SECRET_KEY", 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SSL_VERIFY": { 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SSL_VERIFY", 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-08-19T01:51:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "cross_origin_url": { 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cross_origin_url", 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "crt_file": { 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crt_file", 2024-08-19T01:51:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "debug": { 2024-08-19T01:51:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "debug", 2024-08-19T01:51:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable/disable debug options", 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "jwt_token_ttl": { 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "jwt_token_ttl", 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "28800", 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-08-19T01:51:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "motd": { 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "motd", 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "The message of the day", 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect_resolve_ip_addr": { 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "redirect_resolve_ip_addr", 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8080", 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl_server_port": { 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl_server_port", 2024-08-19T01:51:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8443", 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "redirect", 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect" 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "url_prefix": { 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url_prefix", 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_monitoring": { 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_monitoring", 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "monitor device health metrics", 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "mark_out_threshold": { 2024-08-19T01:51:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mark_out_threshold", 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2419200", 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-08-19T01:51:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": { 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_name", 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "device_health_metrics", 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name of pool in which to store device health metrics", 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "retention_period": { 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retention_period", 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15552000", 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain device health metrics", 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_frequency": { 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_frequency", 2024-08-19T01:51:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to scrape device health metrics", 2024-08-19T01:51:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "self_heal": { 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "self_heal", 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and check device health", 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_threshold": { 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_threshold", 2024-08-19T01:51:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7257600", 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-08-19T01:51:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "diskprediction_local", 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "predict_interval": { 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predict_interval", 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-08-19T01:51:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "predictor_model": { 2024-08-19T01:51:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predictor_model", 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "prophetstor", 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "influx", 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": false, 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "influxdb python module not found", 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: "batch_size": { 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "batch_size", 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5000", 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "database": { 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "database", 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "ceph", 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": { 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hostname", 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server hostname", 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "30", 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "5", 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-08-19T01:51:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "password": { 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "password", 2024-08-19T01:51:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "password of InfluxDB server user", 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "port": { 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "port", 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8086", 2024-08-19T01:51:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server port", 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-08-19T01:51:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "false", 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "threads": { 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threads", 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "32", 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "username": { 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "username", 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "username of InfluxDB server user", 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: "verify_ssl": { 2024-08-19T01:51:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "verify_ssl", 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "true", 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "insights", 2024-08-19T01:51:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "iostat", 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "k8sevents", 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_event_retention_days": { 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ceph_event_retention_days", 2024-08-19T01:51:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7", 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Days to hold ceph event information within local cache", 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "config_check_secs": { 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_check_secs", 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "10", 2024-08-19T01:51:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-08-19T01:51:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "localpool", 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "failure_domain": { 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "failure_domain", 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "host", 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "failure domain for any created local pool", 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": { 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_size", 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default min_size for any created local pool", 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: "num_rep": { 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "num_rep", 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3", 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default replica count for any created local pool", 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": { 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_num", 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "128", 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default pg_num for any created local pool", 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "prefix": { 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prefix", 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name prefix for any created local pool", 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "subtree": { 2024-08-19T01:51:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "subtree", 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "rack", 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "CRUSH level for which to create a local pool", 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mds_autoscaler", 2024-08-19T01:51:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirroring", 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "nfs", 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-08-19T01:51:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "fail_fs": { 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "fail_fs", 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator": { 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rook", 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "test_orchestrator" 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Orchestrator backend", 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_perf_query", 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_support", 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_autoscaler", 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "threshold": { 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threshold", 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1.0", 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "scaling threshold", 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "progress", 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_pg_recovery_event": { 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_pg_recovery_event", 2024-08-19T01:51:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow the module to show pg recovery progress", 2024-08-19T01:51:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "max_completed_events": { 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_completed_events", 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "50", 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "number of past completed events to remember", 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long the module is going to sleep", 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus", 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "cache": { 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cache", 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "exclude_perf_counters": { 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "exclude_perf_counters", 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools": { 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools", 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools_refresh_interval": { 2024-08-19T01:51:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "300", 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_interval": { 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_interval", 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15.0", 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "9283", 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "stale_cache_strategy": { 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stale_cache_strategy", 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "log", 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "default", 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "default", 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "error" 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-08-19T01:51:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_snap_create": { 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_snap_create", 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "mirror_snapshot_schedule": { 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirror_snapshot_schedule", 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "trash_purge_schedule": { 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "trash_purge_schedule", 2024-08-19T01:51:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "restful", 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_auth": { 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_auth", 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-08-19T01:51:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rgw", 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.532 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.533 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.534 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.536 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.537 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.538 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.539 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rook", 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.541 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.542 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.543 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.544 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.545 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_class": { 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "storage_class", 2024-08-19T01:51:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "local", 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-08-19T01:51:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "selftest", 2024-08-19T01:51:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "roption1": { 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption1", 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "roption2": { 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption2", 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "xyz", 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption1": { 2024-08-19T01:51:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption1", 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption2": { 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption2", 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption3": { 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption3", 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption4": { 2024-08-19T01:51:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption4", 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption5": { 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption5", 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption6": { 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption6", 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption7": { 2024-08-19T01:51:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption7", 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "42", 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "testkey": { 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testkey", 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "testlkey": { 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testlkey", 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "testnewline": { 2024-08-19T01:51:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testnewline", 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-08-19T01:51:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_m_granularity": { 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_m_granularity", 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow minute scheduled snapshots", 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "dump_on_update": { 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dump_on_update", 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "dump database to debug log on update", 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stats", 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "status", 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telegraf", 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "address": { 2024-08-19T01:51:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "address", 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15", 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telemetry", 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_basic": { 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_basic", 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share basic cluster information (size, version)", 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_crash": { 2024-08-19T01:51:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_crash", 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-08-19T01:51:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_device": { 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_device", 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_ident": { 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_ident", 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_perf": { 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_perf", 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share various performance metrics of a cluster", 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "contact": { 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "contact", 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "description": { 2024-08-19T01:51:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "description", 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "device_url": { 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_url", 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "24", 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "8", 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "last_opt_revision": { 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "last_opt_revision", 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1", 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard": { 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard", 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard_description": { 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard_description", 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.649 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.650 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.652 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.653 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.653 INFO:teuthology.orchestra.run.smithi064.stdout: "organization": { 2024-08-19T01:51:12.653 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "organization", 2024-08-19T01:51:12.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.653 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: "proxy": { 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "proxy", 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.656 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.657 INFO:teuthology.orchestra.run.smithi064.stdout: "url": { 2024-08-19T01:51:12.657 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url", 2024-08-19T01:51:12.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.657 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.657 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.657 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "test_orchestrator", 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.659 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.661 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.662 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.662 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.662 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.662 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.662 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.663 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.664 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.666 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.667 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.669 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.670 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.672 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_clones": { 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_clones", 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4", 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Number of asynchronous cloner threads", 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "periodic_async_work": { 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "periodic_async_work", 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Periodically check for async work", 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_delay": { 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_delay", 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_no_wait": { 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_no_wait", 2024-08-19T01:51:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix", 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "discovery_interval": { 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "discovery_interval", 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "100", 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "identifier": { 2024-08-19T01:51:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "identifier", 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.695 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.696 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.697 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.698 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.698 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.698 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.698 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.698 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_host": { 2024-08-19T01:51:12.698 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_host", 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.699 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_port": { 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_port", 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10051", 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.701 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_sender": { 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_sender", 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-08-19T01:51:12.702 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: "gid": 24833, 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "y", 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_features": 4540701547738038271, 2024-08-19T01:51:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: "available_modules": [ 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "alerts", 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How frequently to reexamine health status", 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_destination": { 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_destination", 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email address to send alerts to", 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_from_name": { 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_from_name", 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "Ceph", 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email From: name", 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_host": { 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_host", 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP server", 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_password": { 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_password", 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Password to authenticate with", 2024-08-19T01:51:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_port": { 2024-08-19T01:51:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_port", 2024-08-19T01:51:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "465", 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP port", 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_sender": { 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_sender", 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP envelope sender", 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_ssl": { 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_ssl", 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use SSL to connect to SMTP server", 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_user": { 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_user", 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "User to authenticate as", 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "balancer", 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "active": { 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "active", 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically balance PGs across cluster", 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_time": { 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_time", 2024-08-19T01:51:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0000", 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "beginning time of day to automatically balance", 2024-08-19T01:51:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_weekday": { 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_weekday", 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_max_iterations": { 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_max_iterations", 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "25", 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "250", 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_metrics": { 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_metrics", 2024-08-19T01:51:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "pgs,objects,bytes", 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-08-19T01:51:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_step": { 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_step", 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.5", 2024-08-19T01:51:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0.001", 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "0.999", 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "aggressiveness of optimization", 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "end_time": { 2024-08-19T01:51:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_time", 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2359", 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ending time of day to automatically balance", 2024-08-19T01:51:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "end_weekday": { 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_weekday", 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "min_score": { 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_score", 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "upmap", 2024-08-19T01:51:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-compat", 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "none", 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "read", 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap", 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap-read" 2024-08-19T01:51:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Balancer mode", 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_ids": { 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_ids", 2024-08-19T01:51:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-08-19T01:51:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-08-19T01:51:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_deviation": { 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_deviation", 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "deviation below which no optimization is attempted", 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_optimizations": { 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_optimizations", 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm", 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_down_multiplier": { 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_down_multiplier", 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_refresh_rate": { 2024-08-19T01:51:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_refresh_rate", 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "20", 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-08-19T01:51:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_starting_port": { 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_starting_port", 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4721", 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_ptrace": { 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_ptrace", 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_interval": { 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_interval", 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to autotune daemon memory", 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_memory_target_ratio": { 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_memory_target_ratio", 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.7", 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_log_destination": { 2024-08-19T01:51:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm_log_destination", 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "file", 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "file,syslog", 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "syslog" 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "cgroups_split": { 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cgroups_split", 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "config_checks_enabled": { 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_checks_enabled", 2024-08-19T01:51:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "config_dashboard": { 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_dashboard", 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_alertmanager": { 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_alertmanager", 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-08-19T01:51:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_base": { 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_base", 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/ceph", 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Container image name, without the tag", 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_elasticsearch": { 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_elasticsearch", 2024-08-19T01:51:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "elasticsearch container image", 2024-08-19T01:51:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_grafana": { 2024-08-19T01:51:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_grafana", 2024-08-19T01:51:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-08-19T01:51:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_haproxy": { 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_haproxy", 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "HAproxy container image", 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_agent": { 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_agent", 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger agent container image", 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_collector": { 2024-08-19T01:51:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_collector", 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger collector container image", 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_query": { 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_query", 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger query container image", 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.794 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_keepalived": { 2024-08-19T01:51:12.794 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_keepalived", 2024-08-19T01:51:12.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.794 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.794 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Keepalived container image", 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.795 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_loki": { 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_loki", 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.796 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Loki container image", 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_node_exporter": { 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_node_exporter", 2024-08-19T01:51:12.797 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.798 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_nvmeof": { 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_nvmeof", 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.799 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Nvme-of container image", 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_prometheus": { 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_prometheus", 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.801 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_promtail": { 2024-08-19T01:51:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_promtail", 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Promtail container image", 2024-08-19T01:51:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_samba": { 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_samba", 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.804 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Samba/SMB container image", 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.805 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_snmp_gateway": { 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_snmp_gateway", 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.806 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SNMP Gateway container image", 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: "container_init": { 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_init", 2024-08-19T01:51:12.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Run podman/docker with `--init`", 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.808 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: "daemon_cache_timeout": { 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "daemon_cache_timeout", 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.809 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.810 INFO:teuthology.orchestra.run.smithi064.stdout: "default_cephadm_command_timeout": { 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_cephadm_command_timeout", 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "900", 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.811 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: "default_registry": { 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_registry", 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.812 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io", 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.813 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.814 INFO:teuthology.orchestra.run.smithi064.stdout: "device_cache_timeout": { 2024-08-19T01:51:12.814 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_cache_timeout", 2024-08-19T01:51:12.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.814 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.814 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1800", 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache device inventory", 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "device_enhanced_scan": { 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_enhanced_scan", 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.816 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use libstoragemgmt during device scans", 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: "facts_cache_timeout": { 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "facts_cache_timeout", 2024-08-19T01:51:12.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache host facts data", 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "grafana_dashboards_path": { 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "grafana_dashboards_path", 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-08-19T01:51:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: "host_check_interval": { 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "host_check_interval", 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to perform a host check", 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: "hw_monitoring": { 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hw_monitoring", 2024-08-19T01:51:12.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.823 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "inventory_list_all": { 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "inventory_list_all", 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.824 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.825 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.826 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.827 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: "log_refresh_metadata": { 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_refresh_metadata", 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.830 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.830 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.830 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.830 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.830 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.831 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.832 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf": { 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "*", 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "max_count_per_host": { 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_count_per_host", 2024-08-19T01:51:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of daemons per service per host", 2024-08-19T01:51:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd_draining_count": { 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_osd_draining_count", 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "migration_current": { 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "migration_current", 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "internal - do not modify", 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "root", 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm-package", 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "root" 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "mode for remote execution of cephadm", 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "oob_default_addr": { 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "oob_default_addr", 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "169.254.1.1", 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default address for RedFish API (oob management).", 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "prometheus_alerts_path": { 2024-08-19T01:51:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus_alerts_path", 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-08-19T01:51:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_insecure": { 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_insecure", 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-08-19T01:51:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_password": { 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_password", 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_url": { 2024-08-19T01:51:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_url", 2024-08-19T01:51:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_username": { 2024-08-19T01:51:12.855 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_username", 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.856 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: "secure_monitoring_stack": { 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "secure_monitoring_stack", 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: "service_discovery_port": { 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "service_discovery_port", 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.859 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8765", 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "cephadm service discovery port", 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "ssh_config_file": { 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssh_config_file", 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.861 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: "use_agent": { 2024-08-19T01:51:12.862 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_agent", 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.863 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "use_repo_digest": { 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_repo_digest", 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.865 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_failed_host_check": { 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_failed_host_check", 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.866 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if the host check fails", 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.867 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_daemons": { 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_daemons", 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.868 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-08-19T01:51:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_hosts": { 2024-08-19T01:51:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_hosts", 2024-08-19T01:51:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-08-19T01:51:12.870 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crash", 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.871 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.872 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.873 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.873 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.873 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.873 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.873 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.873 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.873 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.874 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.874 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.874 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.874 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.874 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.874 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.875 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.877 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.878 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.879 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.880 INFO:teuthology.orchestra.run.smithi064.stdout: "retain_interval": { 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retain_interval", 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "31536000", 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.881 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain crashes before pruning them", 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_recent_interval": { 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_recent_interval", 2024-08-19T01:51:12.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1209600", 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "time interval in which to warn about recent crashes", 2024-08-19T01:51:12.883 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dashboard", 2024-08-19T01:51:12.884 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.885 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.886 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_HOST": { 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_HOST", 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.887 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-08-19T01:51:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.889 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_ENABLED": { 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_ENABLED", 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.890 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.892 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "ENABLE_BROWSABLE_API": { 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ENABLE_BROWSABLE_API", 2024-08-19T01:51:12.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.894 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.896 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.897 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-08-19T01:51:12.898 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.900 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.902 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_NFS": { 2024-08-19T01:51:12.902 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-08-19T01:51:12.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.902 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.902 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.902 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RBD": { 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.904 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RGW": { 2024-08-19T01:51:12.905 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.906 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.907 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.908 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_PASSWORD": { 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_PASSWORD", 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:12.909 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.910 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.911 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_URL": { 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_URL", 2024-08-19T01:51:12.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.913 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_USERNAME": { 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_USERNAME", 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.914 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.916 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-08-19T01:51:12.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.918 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.918 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.918 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.918 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.918 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.918 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.919 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.920 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ISSUE_TRACKER_API_KEY": { 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.921 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_HOST": { 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_HOST", 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-08-19T01:51:12.924 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.925 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.928 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.929 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.930 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.931 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.932 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.933 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.935 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-08-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_ENABLED": { 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_ENABLED", 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.940 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-08-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.942 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.943 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8", 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.944 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: "REST_REQUESTS_TIMEOUT": { 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "45", 2024-08-19T01:51:12.945 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.946 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ACCESS_KEY": { 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ACCESS_KEY", 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.947 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.948 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SECRET_KEY": { 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SECRET_KEY", 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.950 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SSL_VERIFY": { 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SSL_VERIFY", 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.952 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-08-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.954 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "cross_origin_url": { 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cross_origin_url", 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.959 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: "crt_file": { 2024-08-19T01:51:12.960 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crt_file", 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.961 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.962 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.962 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.962 INFO:teuthology.orchestra.run.smithi064.stdout: "debug": { 2024-08-19T01:51:12.962 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "debug", 2024-08-19T01:51:12.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable/disable debug options", 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: "jwt_token_ttl": { 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "jwt_token_ttl", 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "28800", 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.968 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.970 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.970 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.970 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.970 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.970 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.970 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.973 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:12.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.975 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "motd": { 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "motd", 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "The message of the day", 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect_resolve_ip_addr": { 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "redirect_resolve_ip_addr", 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8080", 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl_server_port": { 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl_server_port", 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8443", 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-08-19T01:51:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "redirect", 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect" 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-08-19T01:51:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "url_prefix": { 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url_prefix", 2024-08-19T01:51:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_monitoring": { 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_monitoring", 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "monitor device health metrics", 2024-08-19T01:51:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "mark_out_threshold": { 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mark_out_threshold", 2024-08-19T01:51:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2419200", 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": { 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_name", 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "device_health_metrics", 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name of pool in which to store device health metrics", 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "retention_period": { 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retention_period", 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15552000", 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain device health metrics", 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_frequency": { 2024-08-19T01:51:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_frequency", 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to scrape device health metrics", 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "self_heal": { 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "self_heal", 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-08-19T01:51:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and check device health", 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_threshold": { 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_threshold", 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7257600", 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "diskprediction_local", 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "predict_interval": { 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predict_interval", 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "predictor_model": { 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predictor_model", 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "prophetstor", 2024-08-19T01:51:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "influx", 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": false, 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "influxdb python module not found", 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "batch_size": { 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "batch_size", 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5000", 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "database": { 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "database", 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "ceph", 2024-08-19T01:51:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-08-19T01:51:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": { 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hostname", 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server hostname", 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "30", 2024-08-19T01:51:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "5", 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "password": { 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "password", 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "password of InfluxDB server user", 2024-08-19T01:51:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "port": { 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "port", 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8086", 2024-08-19T01:51:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server port", 2024-08-19T01:51:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "false", 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-08-19T01:51:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "threads": { 2024-08-19T01:51:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threads", 2024-08-19T01:51:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "32", 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "username": { 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "username", 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "username of InfluxDB server user", 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "verify_ssl": { 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "verify_ssl", 2024-08-19T01:51:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "true", 2024-08-19T01:51:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "insights", 2024-08-19T01:51:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "iostat", 2024-08-19T01:51:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "k8sevents", 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_event_retention_days": { 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ceph_event_retention_days", 2024-08-19T01:51:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7", 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Days to hold ceph event information within local cache", 2024-08-19T01:51:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "config_check_secs": { 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_check_secs", 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "10", 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "localpool", 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "failure_domain": { 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "failure_domain", 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "host", 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "failure domain for any created local pool", 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": { 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_size", 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default min_size for any created local pool", 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-08-19T01:51:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "num_rep": { 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "num_rep", 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3", 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default replica count for any created local pool", 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": { 2024-08-19T01:51:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_num", 2024-08-19T01:51:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "128", 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default pg_num for any created local pool", 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "prefix": { 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prefix", 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name prefix for any created local pool", 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "subtree": { 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "subtree", 2024-08-19T01:51:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "rack", 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "CRUSH level for which to create a local pool", 2024-08-19T01:51:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mds_autoscaler", 2024-08-19T01:51:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirroring", 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "nfs", 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-08-19T01:51:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "fail_fs": { 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "fail_fs", 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator": { 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "rook", 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "test_orchestrator" 2024-08-19T01:51:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Orchestrator backend", 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_perf_query", 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_support", 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_autoscaler", 2024-08-19T01:51:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "threshold": { 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threshold", 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1.0", 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "scaling threshold", 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "progress", 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_pg_recovery_event": { 2024-08-19T01:51:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_pg_recovery_event", 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow the module to show pg recovery progress", 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "max_completed_events": { 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_completed_events", 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "50", 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "number of past completed events to remember", 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long the module is going to sleep", 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus", 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "cache": { 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cache", 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "exclude_perf_counters": { 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "exclude_perf_counters", 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools": { 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools", 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools_refresh_interval": { 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "300", 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_interval": { 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_interval", 2024-08-19T01:51:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15.0", 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "9283", 2024-08-19T01:51:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "stale_cache_strategy": { 2024-08-19T01:51:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stale_cache_strategy", 2024-08-19T01:51:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "log", 2024-08-19T01:51:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-08-19T01:51:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "default", 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "default", 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: "error" 2024-08-19T01:51:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-08-19T01:51:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_snap_create": { 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_snap_create", 2024-08-19T01:51:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: "mirror_snapshot_schedule": { 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirror_snapshot_schedule", 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.240 INFO:teuthology.orchestra.run.smithi064.stdout: "trash_purge_schedule": { 2024-08-19T01:51:13.240 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "trash_purge_schedule", 2024-08-19T01:51:13.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.240 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.240 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.240 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "restful", 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_auth": { 2024-08-19T01:51:13.242 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_auth", 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.243 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.244 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.245 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.246 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.247 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.249 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.251 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.251 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.251 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.251 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.251 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.251 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.251 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.252 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.253 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:13.254 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.255 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.256 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.257 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rgw", 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.258 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.259 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.260 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.261 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.261 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.261 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.261 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.263 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.264 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.265 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.265 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.265 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.265 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.265 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.265 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.266 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.267 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rook", 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.269 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.270 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.271 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.277 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_class": { 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "storage_class", 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.278 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "local", 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.279 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "selftest", 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.280 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.281 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.282 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.284 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.285 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.286 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.287 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.288 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "roption1": { 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption1", 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.289 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.290 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: "roption2": { 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption2", 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.291 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "xyz", 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption1": { 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption1", 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption2": { 2024-08-19T01:51:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption2", 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption3": { 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption3", 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption4": { 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption4", 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption5": { 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption5", 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption6": { 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption6", 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption7": { 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption7", 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "42", 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "testkey": { 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testkey", 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "testlkey": { 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testlkey", 2024-08-19T01:51:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "testnewline": { 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testnewline", 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-08-19T01:51:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_m_granularity": { 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_m_granularity", 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow minute scheduled snapshots", 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "dump_on_update": { 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dump_on_update", 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "dump database to debug log on update", 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stats", 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "status", 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telegraf", 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "address": { 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "address", 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15", 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telemetry", 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_basic": { 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_basic", 2024-08-19T01:51:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share basic cluster information (size, version)", 2024-08-19T01:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_crash": { 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_crash", 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_device": { 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_device", 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_ident": { 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_ident", 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_perf": { 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_perf", 2024-08-19T01:51:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share various performance metrics of a cluster", 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "contact": { 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "contact", 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "description": { 2024-08-19T01:51:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "description", 2024-08-19T01:51:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "device_url": { 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_url", 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-08-19T01:51:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "24", 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "8", 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "last_opt_revision": { 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "last_opt_revision", 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1", 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard": { 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard", 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard_description": { 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard_description", 2024-08-19T01:51:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "organization": { 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "organization", 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "proxy": { 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "proxy", 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "url": { 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url", 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "test_orchestrator", 2024-08-19T01:51:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-08-19T01:51:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_clones": { 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_clones", 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4", 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Number of asynchronous cloner threads", 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "periodic_async_work": { 2024-08-19T01:51:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "periodic_async_work", 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Periodically check for async work", 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_delay": { 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_delay", 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_no_wait": { 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_no_wait", 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix", 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "discovery_interval": { 2024-08-19T01:51:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "discovery_interval", 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "100", 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "identifier": { 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "identifier", 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.428 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_host": { 2024-08-19T01:51:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_host", 2024-08-19T01:51:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_port": { 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_port", 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10051", 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_sender": { 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_sender", 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-08-19T01:51:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "modules": [ 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "dashboard", 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "iostat", 2024-08-19T01:51:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "nfs", 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "restful", 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_schedule" 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "available_modules": [ 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "alerts", 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How frequently to reexamine health status", 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_destination": { 2024-08-19T01:51:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_destination", 2024-08-19T01:51:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email address to send alerts to", 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_from_name": { 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_from_name", 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "Ceph", 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email From: name", 2024-08-19T01:51:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_host": { 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_host", 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP server", 2024-08-19T01:51:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_password": { 2024-08-19T01:51:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_password", 2024-08-19T01:51:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Password to authenticate with", 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_port": { 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_port", 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "465", 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP port", 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_sender": { 2024-08-19T01:51:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_sender", 2024-08-19T01:51:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP envelope sender", 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_ssl": { 2024-08-19T01:51:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_ssl", 2024-08-19T01:51:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use SSL to connect to SMTP server", 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_user": { 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_user", 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "User to authenticate as", 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "balancer", 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "active": { 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "active", 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.471 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.471 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.471 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.471 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.471 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically balance PGs across cluster", 2024-08-19T01:51:13.471 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_time": { 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_time", 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.472 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0000", 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "beginning time of day to automatically balance", 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_weekday": { 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_weekday", 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-08-19T01:51:13.474 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_max_iterations": { 2024-08-19T01:51:13.475 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_max_iterations", 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "25", 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "250", 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.476 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_metrics": { 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_metrics", 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.477 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "pgs,objects,bytes", 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-08-19T01:51:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_step": { 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_step", 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.5", 2024-08-19T01:51:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0.001", 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "0.999", 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "aggressiveness of optimization", 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "end_time": { 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_time", 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2359", 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ending time of day to automatically balance", 2024-08-19T01:51:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-08-19T01:51:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "end_weekday": { 2024-08-19T01:51:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_weekday", 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.490 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "min_score": { 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_score", 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-08-19T01:51:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-08-19T01:51:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "upmap", 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-compat", 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "none", 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "read", 2024-08-19T01:51:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap", 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap-read" 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Balancer mode", 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_ids": { 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_ids", 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_deviation": { 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_deviation", 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:13.501 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "deviation below which no optimization is attempted", 2024-08-19T01:51:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-08-19T01:51:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_optimizations": { 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_optimizations", 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.503 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm", 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_down_multiplier": { 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_down_multiplier", 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.505 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.506 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.506 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-08-19T01:51:13.506 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.506 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.506 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.506 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-08-19T01:51:13.506 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_refresh_rate": { 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_refresh_rate", 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.507 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "20", 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.509 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_starting_port": { 2024-08-19T01:51:13.509 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_starting_port", 2024-08-19T01:51:13.509 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.509 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.509 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.509 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4721", 2024-08-19T01:51:13.509 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_ptrace": { 2024-08-19T01:51:13.510 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_ptrace", 2024-08-19T01:51:13.511 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.511 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.511 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.511 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.511 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.511 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.511 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_interval": { 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_interval", 2024-08-19T01:51:13.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to autotune daemon memory", 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.514 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.514 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_memory_target_ratio": { 2024-08-19T01:51:13.514 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_memory_target_ratio", 2024-08-19T01:51:13.514 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.514 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.514 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.7", 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.515 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_log_destination": { 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm_log_destination", 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.516 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: "file", 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: "file,syslog", 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: "syslog" 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-08-19T01:51:13.517 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: "cgroups_split": { 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cgroups_split", 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.518 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.520 INFO:teuthology.orchestra.run.smithi064.stdout: "config_checks_enabled": { 2024-08-19T01:51:13.520 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_checks_enabled", 2024-08-19T01:51:13.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.520 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.520 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.520 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "config_dashboard": { 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_dashboard", 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_alertmanager": { 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_alertmanager", 2024-08-19T01:51:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.524 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.524 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-08-19T01:51:13.524 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.524 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.524 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.524 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_base": { 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_base", 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.525 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/ceph", 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Container image name, without the tag", 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.527 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.527 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_elasticsearch": { 2024-08-19T01:51:13.527 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_elasticsearch", 2024-08-19T01:51:13.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.527 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "elasticsearch container image", 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.528 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_grafana": { 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_grafana", 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-08-19T01:51:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_haproxy": { 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_haproxy", 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.531 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "HAproxy container image", 2024-08-19T01:51:13.532 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.532 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.532 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.532 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_agent": { 2024-08-19T01:51:13.532 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_agent", 2024-08-19T01:51:13.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-08-19T01:51:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger agent container image", 2024-08-19T01:51:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_collector": { 2024-08-19T01:51:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_collector", 2024-08-19T01:51:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger collector container image", 2024-08-19T01:51:13.535 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_query": { 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_query", 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.536 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.537 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-08-19T01:51:13.537 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.537 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.537 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.537 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger query container image", 2024-08-19T01:51:13.537 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.537 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_keepalived": { 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_keepalived", 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.538 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Keepalived container image", 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_loki": { 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_loki", 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.540 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.541 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Loki container image", 2024-08-19T01:51:13.541 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.541 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.541 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.541 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_node_exporter": { 2024-08-19T01:51:13.541 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_node_exporter", 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.542 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_nvmeof": { 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_nvmeof", 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.544 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-08-19T01:51:13.544 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.544 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.544 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.544 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Nvme-of container image", 2024-08-19T01:51:13.544 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.544 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_prometheus": { 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_prometheus", 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.545 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_promtail": { 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_promtail", 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Promtail container image", 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_samba": { 2024-08-19T01:51:13.548 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_samba", 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.549 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Samba/SMB container image", 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_snmp_gateway": { 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_snmp_gateway", 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SNMP Gateway container image", 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.551 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: "container_init": { 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_init", 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.552 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.553 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.553 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.553 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.553 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Run podman/docker with `--init`", 2024-08-19T01:51:13.553 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.553 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.553 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: "daemon_cache_timeout": { 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "daemon_cache_timeout", 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:13.554 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.555 INFO:teuthology.orchestra.run.smithi064.stdout: "default_cephadm_command_timeout": { 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_cephadm_command_timeout", 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "900", 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.556 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: "default_registry": { 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_registry", 2024-08-19T01:51:13.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io", 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-08-19T01:51:13.558 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: "device_cache_timeout": { 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_cache_timeout", 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1800", 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache device inventory", 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.560 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: "device_enhanced_scan": { 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_enhanced_scan", 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.561 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use libstoragemgmt during device scans", 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.562 INFO:teuthology.orchestra.run.smithi064.stdout: "facts_cache_timeout": { 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "facts_cache_timeout", 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.563 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache host facts data", 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "grafana_dashboards_path": { 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "grafana_dashboards_path", 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.565 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "host_check_interval": { 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "host_check_interval", 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.566 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to perform a host check", 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "hw_monitoring": { 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hw_monitoring", 2024-08-19T01:51:13.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "inventory_list_all": { 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "inventory_list_all", 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.569 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.570 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.571 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.572 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "log_refresh_metadata": { 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_refresh_metadata", 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.573 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.574 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.575 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.576 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.577 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.578 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.579 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.580 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf": { 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.581 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.582 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-08-19T01:51:13.582 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.582 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.582 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.582 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-08-19T01:51:13.582 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "*", 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-08-19T01:51:13.583 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: "max_count_per_host": { 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_count_per_host", 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of daemons per service per host", 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.585 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd_draining_count": { 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_osd_draining_count", 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.586 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.587 INFO:teuthology.orchestra.run.smithi064.stdout: "migration_current": { 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "migration_current", 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.588 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "internal - do not modify", 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.589 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "root", 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm-package", 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: "root" 2024-08-19T01:51:13.590 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "mode for remote execution of cephadm", 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: "oob_default_addr": { 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "oob_default_addr", 2024-08-19T01:51:13.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "169.254.1.1", 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default address for RedFish API (oob management).", 2024-08-19T01:51:13.592 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: "prometheus_alerts_path": { 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus_alerts_path", 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.594 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_insecure": { 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_insecure", 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.595 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.596 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_password": { 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_password", 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.597 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.598 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-08-19T01:51:13.598 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.598 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.598 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.598 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_url": { 2024-08-19T01:51:13.598 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_url", 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.599 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_username": { 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_username", 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.600 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.601 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: "secure_monitoring_stack": { 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "secure_monitoring_stack", 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.602 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "service_discovery_port": { 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "service_discovery_port", 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8765", 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.604 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "cephadm service discovery port", 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: "ssh_config_file": { 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssh_config_file", 2024-08-19T01:51:13.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.606 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.606 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.606 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.606 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.606 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.606 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.606 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: "use_agent": { 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_agent", 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.607 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.608 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "use_repo_digest": { 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_repo_digest", 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.609 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.610 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_failed_host_check": { 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_failed_host_check", 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.611 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if the host check fails", 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_daemons": { 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_daemons", 2024-08-19T01:51:13.612 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.613 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.613 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.613 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.613 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.613 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.613 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_hosts": { 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_hosts", 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.616 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crash", 2024-08-19T01:51:13.616 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.616 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.617 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.618 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.619 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.620 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.621 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.622 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.623 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.624 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "retain_interval": { 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retain_interval", 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "31536000", 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain crashes before pruning them", 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.626 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_recent_interval": { 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_recent_interval", 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1209600", 2024-08-19T01:51:13.627 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.628 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.628 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.628 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "time interval in which to warn about recent crashes", 2024-08-19T01:51:13.628 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.628 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.628 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dashboard", 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.629 INFO:teuthology.orchestra.run.smithi064.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-08-19T01:51:13.630 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-08-19T01:51:13.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.630 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.630 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.630 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.630 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.631 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.631 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.631 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.631 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.631 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.632 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_HOST": { 2024-08-19T01:51:13.632 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_HOST", 2024-08-19T01:51:13.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.632 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.632 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.632 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.633 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.634 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_ENABLED": { 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_ENABLED", 2024-08-19T01:51:13.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.636 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.637 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.638 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.639 INFO:teuthology.orchestra.run.smithi064.stdout: "ENABLE_BROWSABLE_API": { 2024-08-19T01:51:13.639 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ENABLE_BROWSABLE_API", 2024-08-19T01:51:13.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.639 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.639 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.641 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-08-19T01:51:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_NFS": { 2024-08-19T01:51:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RBD": { 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RGW": { 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_PASSWORD": { 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_PASSWORD", 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-08-19T01:51:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_URL": { 2024-08-19T01:51:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_URL", 2024-08-19T01:51:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_USERNAME": { 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_USERNAME", 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.662 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-08-19T01:51:13.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.664 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.665 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "ISSUE_TRACKER_API_KEY": { 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.667 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_HOST": { 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_HOST", 2024-08-19T01:51:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.669 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.671 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.672 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.673 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.674 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.675 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.676 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.677 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.678 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.679 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.680 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.682 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.683 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_ENABLED": { 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_ENABLED", 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.684 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-08-19T01:51:13.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.687 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8", 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: "REST_REQUESTS_TIMEOUT": { 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-08-19T01:51:13.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "45", 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.691 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ACCESS_KEY": { 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ACCESS_KEY", 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.692 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.693 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-08-19T01:51:13.694 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.695 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SECRET_KEY": { 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SECRET_KEY", 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.696 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SSL_VERIFY": { 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SSL_VERIFY", 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.697 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.698 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:13.699 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.700 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.701 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-08-19T01:51:13.702 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.703 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: "cross_origin_url": { 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cross_origin_url", 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.704 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "crt_file": { 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crt_file", 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.706 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "debug": { 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "debug", 2024-08-19T01:51:13.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable/disable debug options", 2024-08-19T01:51:13.708 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "jwt_token_ttl": { 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "jwt_token_ttl", 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.709 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "28800", 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.712 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.713 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.714 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.715 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.716 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.717 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.718 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.719 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: "motd": { 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "motd", 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "The message of the day", 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect_resolve_ip_addr": { 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "redirect_resolve_ip_addr", 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.723 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:13.724 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.725 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.726 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8080", 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.727 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.728 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl_server_port": { 2024-08-19T01:51:13.729 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl_server_port", 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8443", 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.730 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.731 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "redirect", 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect" 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.732 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.734 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: "url_prefix": { 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url_prefix", 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.735 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_monitoring": { 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_monitoring", 2024-08-19T01:51:13.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "monitor device health metrics", 2024-08-19T01:51:13.738 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.739 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.740 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.741 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.742 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "mark_out_threshold": { 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mark_out_threshold", 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2419200", 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": { 2024-08-19T01:51:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_name", 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "device_health_metrics", 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name of pool in which to store device health metrics", 2024-08-19T01:51:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "retention_period": { 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retention_period", 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15552000", 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain device health metrics", 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_frequency": { 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_frequency", 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-08-19T01:51:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to scrape device health metrics", 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "self_heal": { 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "self_heal", 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and check device health", 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_threshold": { 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_threshold", 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7257600", 2024-08-19T01:51:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-08-19T01:51:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "diskprediction_local", 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "predict_interval": { 2024-08-19T01:51:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predict_interval", 2024-08-19T01:51:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-08-19T01:51:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "predictor_model": { 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predictor_model", 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "prophetstor", 2024-08-19T01:51:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "influx", 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": false, 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "influxdb python module not found", 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "batch_size": { 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "batch_size", 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5000", 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "database": { 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "database", 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "ceph", 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": { 2024-08-19T01:51:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hostname", 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server hostname", 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "30", 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "5", 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "password": { 2024-08-19T01:51:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "password", 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "password of InfluxDB server user", 2024-08-19T01:51:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "port": { 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "port", 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8086", 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server port", 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "false", 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "threads": { 2024-08-19T01:51:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threads", 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "32", 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "username": { 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "username", 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "username of InfluxDB server user", 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "verify_ssl": { 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "verify_ssl", 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "true", 2024-08-19T01:51:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "insights", 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "iostat", 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "k8sevents", 2024-08-19T01:51:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_event_retention_days": { 2024-08-19T01:51:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ceph_event_retention_days", 2024-08-19T01:51:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7", 2024-08-19T01:51:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Days to hold ceph event information within local cache", 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "config_check_secs": { 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_check_secs", 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "10", 2024-08-19T01:51:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-08-19T01:51:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "localpool", 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "failure_domain": { 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "failure_domain", 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "host", 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "failure domain for any created local pool", 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": { 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_size", 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default min_size for any created local pool", 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "num_rep": { 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "num_rep", 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3", 2024-08-19T01:51:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default replica count for any created local pool", 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": { 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_num", 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "128", 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default pg_num for any created local pool", 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "prefix": { 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prefix", 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name prefix for any created local pool", 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "subtree": { 2024-08-19T01:51:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "subtree", 2024-08-19T01:51:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "rack", 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "CRUSH level for which to create a local pool", 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mds_autoscaler", 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirroring", 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "nfs", 2024-08-19T01:51:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "fail_fs": { 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "fail_fs", 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator": { 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "rook", 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "test_orchestrator" 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Orchestrator backend", 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_perf_query", 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_support", 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_autoscaler", 2024-08-19T01:51:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "threshold": { 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threshold", 2024-08-19T01:51:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1.0", 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "scaling threshold", 2024-08-19T01:51:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "progress", 2024-08-19T01:51:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_pg_recovery_event": { 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_pg_recovery_event", 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow the module to show pg recovery progress", 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "max_completed_events": { 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_completed_events", 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "50", 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "number of past completed events to remember", 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-08-19T01:51:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long the module is going to sleep", 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus", 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "cache": { 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cache", 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "exclude_perf_counters": { 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "exclude_perf_counters", 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.948 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools": { 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools", 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools_refresh_interval": { 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "300", 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_interval": { 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_interval", 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15.0", 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "9283", 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "stale_cache_strategy": { 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stale_cache_strategy", 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "log", 2024-08-19T01:51:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "default", 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "default", 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "error" 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-08-19T01:51:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_snap_create": { 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_snap_create", 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "mirror_snapshot_schedule": { 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirror_snapshot_schedule", 2024-08-19T01:51:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "trash_purge_schedule": { 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "trash_purge_schedule", 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "restful", 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_auth": { 2024-08-19T01:51:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_auth", 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rgw", 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.002 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.003 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.004 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.005 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rook", 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.008 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.009 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.010 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.012 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.013 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.014 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.016 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_class": { 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "storage_class", 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "local", 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "selftest", 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.019 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.020 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.021 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.023 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.024 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.025 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.027 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: "roption1": { 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption1", 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.028 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "roption2": { 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption2", 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "xyz", 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.030 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption1": { 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption1", 2024-08-19T01:51:14.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.032 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption2": { 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption2", 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.033 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption3": { 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption3", 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.035 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption4": { 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption4", 2024-08-19T01:51:14.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.037 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption5": { 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption5", 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption6": { 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption6", 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.040 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption7": { 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption7", 2024-08-19T01:51:14.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "42", 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.042 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: "testkey": { 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testkey", 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "testlkey": { 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testlkey", 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.045 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "testnewline": { 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testnewline", 2024-08-19T01:51:14.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.048 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_m_granularity": { 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_m_granularity", 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.049 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow minute scheduled snapshots", 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.050 INFO:teuthology.orchestra.run.smithi064.stdout: "dump_on_update": { 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dump_on_update", 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.051 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "dump database to debug log on update", 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.053 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.055 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.056 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.057 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.058 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.059 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stats", 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.061 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.062 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.063 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.064 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.065 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.066 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.067 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.069 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "status", 2024-08-19T01:51:14.070 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.071 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.072 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.073 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.074 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.075 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.076 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.078 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telegraf", 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "address": { 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "address", 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.080 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.081 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15", 2024-08-19T01:51:14.082 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.083 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.084 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.085 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.087 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.088 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.089 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telemetry", 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.092 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_basic": { 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_basic", 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.093 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share basic cluster information (size, version)", 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_crash": { 2024-08-19T01:51:14.094 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_crash", 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-08-19T01:51:14.095 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_device": { 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_device", 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.098 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_ident": { 2024-08-19T01:51:14.098 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_ident", 2024-08-19T01:51:14.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.098 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.098 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.098 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_perf": { 2024-08-19T01:51:14.099 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_perf", 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share various performance metrics of a cluster", 2024-08-19T01:51:14.100 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: "contact": { 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "contact", 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "description": { 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "description", 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.103 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: "device_url": { 2024-08-19T01:51:14.104 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_url", 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.105 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "24", 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "8", 2024-08-19T01:51:14.108 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "last_opt_revision": { 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "last_opt_revision", 2024-08-19T01:51:14.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1", 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.110 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard": { 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard", 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.111 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard_description": { 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard_description", 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.113 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.115 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.116 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.117 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.118 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.119 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.120 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.122 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: "organization": { 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "organization", 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.123 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.124 INFO:teuthology.orchestra.run.smithi064.stdout: "proxy": { 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "proxy", 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: "url": { 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url", 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.126 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "test_orchestrator", 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.128 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.129 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.130 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.131 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.133 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.134 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.135 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.136 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-08-19T01:51:14.137 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.138 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.139 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.141 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.143 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.143 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.143 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.143 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.143 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.143 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.143 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.145 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_clones": { 2024-08-19T01:51:14.146 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_clones", 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4", 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.147 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Number of asynchronous cloner threads", 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "periodic_async_work": { 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "periodic_async_work", 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Periodically check for async work", 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.149 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_delay": { 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_delay", 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-08-19T01:51:14.150 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_no_wait": { 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_no_wait", 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.152 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.153 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix", 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "discovery_interval": { 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "discovery_interval", 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.154 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "100", 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "identifier": { 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "identifier", 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.156 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-08-19T01:51:14.157 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.158 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.160 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.162 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-08-19T01:51:14.163 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-08-19T01:51:14.164 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-08-19T01:51:14.166 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.167 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_host": { 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_host", 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.168 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_port": { 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_port", 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-08-19T01:51:14.169 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10051", 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.170 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_sender": { 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_sender", 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-08-19T01:51:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: "services": { 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: "dashboard": "https://172.21.15.64:8443/" 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: "always_on_modules": { 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus": [ 2024-08-19T01:51:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-08-19T01:51:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific": [ 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-08-19T01:51:14.175 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy": [ 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-08-19T01:51:14.176 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-08-19T01:51:14.177 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "reef": [ 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-08-19T01:51:14.178 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "squid": [ 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-08-19T01:51:14.179 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: "last_failure_osd_epoch": 94, 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: "active_clients": [ 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-08-19T01:51:14.181 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2932119682 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "libcephsqlite", 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-08-19T01:51:14.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 727380122 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.183 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3370585448 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1718849873 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.185 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-08-19T01:51:14.186 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3210403928 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-08-19T01:51:14.187 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-08-19T01:51:14.188 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1866560470 2024-08-19T01:51:14.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:14.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-08-19T01:51:14.188 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:51:14.190 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:12 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/1146510980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-08-19T01:51:14.190 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:12 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2796795582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-08-19T01:51:14.190 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:12 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2323108208' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-08-19T01:51:14.190 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:13 smithi161 ceph-mon[21757]: pgmap v565: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 12 MiB/s wr, 406 op/s 2024-08-19T01:51:14.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:12 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/1146510980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-08-19T01:51:14.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:12 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2796795582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-08-19T01:51:14.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:12 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2323108208' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-08-19T01:51:14.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:13 smithi123 ceph-mon[21829]: pgmap v565: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 12 MiB/s wr, 406 op/s 2024-08-19T01:51:14.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f13180a2d00 msgr2=0x7f13180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f13180a2d00 0x7f13180a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f132000bbe0 tx=0x7f132000b390 comp rx=0 tx=0).stop 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 msgr2=0x7f132c19c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c19c9d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f132402ae70 tx=0x7f132402f5d0 comp rx=0 tx=0).stop 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 shutdown_connections 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f13180a2d00 0x7f13180a51c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f132c10bdb0 0x7f132c19c9d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f132c1032b0 0x7f132c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 --2- 172.21.15.64:0/2323108208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c102990 0x7f132c198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:14.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 >> 172.21.15.64:0/2323108208 conn(0x7f132c0fc400 msgr2=0x7f132c0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:14.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 shutdown_connections 2024-08-19T01:51:14.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:11.990+0000 7f133414d640 1 -- 172.21.15.64:0/2323108208 wait complete. 2024-08-19T01:51:14.200 DEBUG:tasks.check_counter:Getting stats from a 2024-08-19T01:51:14.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.a perf dump 2024-08-19T01:51:14.200 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:12 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/1146510980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-08-19T01:51:14.201 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:12 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2796795582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-08-19T01:51:14.201 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:12 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2323108208' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-08-19T01:51:14.201 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:13 smithi064 ceph-mon[18515]: pgmap v565: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 12 MiB/s wr, 406 op/s 2024-08-19T01:51:14.475 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:15.052 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:51:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 0, 2024-08-19T01:51:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2, 2024-08-19T01:51:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 0, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 0, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001087216, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000000000, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000096404, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 344, 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 331, 2024-08-19T01:51:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1617616, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 203888, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.051983250, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.019731220, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.044602811, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 331, 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.019073056, 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000057622 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1617616, 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 203888 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 3, 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 253, 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 480, 2024-08-19T01:51:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 279872, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.017509005, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.011941113, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.002567181, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 250, 2024-08-19T01:51:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.017485150, 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000069940 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 480, 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 279872 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 148670, 2024-08-19T01:51:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 30, 2024-08-19T01:51:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-08-19T01:51:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-08-19T01:51:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 343, 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1566571, 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 343, 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 343, 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1566571, 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:15.071 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:51:15.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:14 smithi123 ceph-mon[21829]: osdmap e118: 12 total, 12 up, 12 in 2024-08-19T01:51:15.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:14 smithi161 ceph-mon[21757]: osdmap e118: 12 total, 12 up, 12 in 2024-08-19T01:51:15.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:14 smithi064 ceph-mon[18515]: osdmap e118: 12 total, 12 up, 12 in 2024-08-19T01:51:15.730 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-08-19T01:51:15.730 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-08-19T01:51:15.730 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-08-19T01:51:15.730 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-08-19T01:51:15.730 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-08-19T01:51:15.731 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-08-19T01:51:15.731 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-08-19T01:51:15.731 DEBUG:tasks.check_counter:Getting stats from d 2024-08-19T01:51:15.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.d perf dump 2024-08-19T01:51:15.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:15 smithi064 ceph-mon[18515]: pgmap v567: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 4.3 MiB/s wr, 140 op/s 2024-08-19T01:51:15.988 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:16.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:15 smithi123 ceph-mon[21829]: pgmap v567: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 4.3 MiB/s wr, 140 op/s 2024-08-19T01:51:16.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:15 smithi161 ceph-mon[21757]: pgmap v567: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 4.3 MiB/s wr, 140 op/s 2024-08-19T01:51:16.501 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:51:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 0, 2024-08-19T01:51:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2, 2024-08-19T01:51:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 0, 2024-08-19T01:51:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 0, 2024-08-19T01:51:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-08-19T01:51:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001109563, 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000000000, 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000229327, 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-08-19T01:51:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 7, 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 259, 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 3552, 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 281024, 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.019993916, 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.012974186, 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.004286647, 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 257, 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.020092443, 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000078180 2024-08-19T01:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 281024 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 345, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 334, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1615904, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 206640, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.052343575, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.020263829, 2024-08-19T01:51:16.508 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.044889508, 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 333, 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.015146489, 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000045484 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:16.509 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1615904, 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 206640 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-08-19T01:51:16.510 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:16.511 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:16.512 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:16.512 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:16.513 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:16.513 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:16.513 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:16.513 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:16.515 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:16.515 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:16.515 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:16.515 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:16.515 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:16.516 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:16.516 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:16.516 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:16.516 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:16.517 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:16.517 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:16.517 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:16.517 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-08-19T01:51:16.518 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:16.518 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:16.518 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:16.518 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 148748, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 32, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:16.519 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:16.520 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:16.520 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:16.520 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-08-19T01:51:16.520 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-08-19T01:51:16.520 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:16.520 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:16.520 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:16.521 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 348, 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1567221, 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 348, 2024-08-19T01:51:16.522 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 348, 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1567221, 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:16.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:16.524 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:51:17.500 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-08-19T01:51:17.500 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-08-19T01:51:17.500 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-08-19T01:51:17.500 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-08-19T01:51:17.501 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-08-19T01:51:17.501 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-08-19T01:51:17.501 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-08-19T01:51:17.501 DEBUG:tasks.check_counter:Getting stats from g 2024-08-19T01:51:17.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.g perf dump 2024-08-19T01:51:17.762 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:18.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:17 smithi123 ceph-mon[21829]: pgmap v568: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-08-19T01:51:18.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:17 smithi064 ceph-mon[18515]: pgmap v568: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-08-19T01:51:18.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:17 smithi161 ceph-mon[21757]: pgmap v568: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 4.0 MiB/s wr, 152 op/s 2024-08-19T01:51:18.285 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:51:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 0, 2024-08-19T01:51:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2, 2024-08-19T01:51:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 0, 2024-08-19T01:51:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 0, 2024-08-19T01:51:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-08-19T01:51:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-08-19T01:51:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001285278, 2024-08-19T01:51:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000000000, 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000268799, 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 7, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 256, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 3552, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 280496, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.020208377, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.013110912, 2024-08-19T01:51:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.004215114, 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 254, 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.020721936, 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000081582 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 280496 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 341, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 330, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1614816, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 204080, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.052173002, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.020898150, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.043684125, 2024-08-19T01:51:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 329, 2024-08-19T01:51:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.015998822, 2024-08-19T01:51:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000048628 2024-08-19T01:51:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1614816, 2024-08-19T01:51:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 204080 2024-08-19T01:51:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-08-19T01:51:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 148731, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 31, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-08-19T01:51:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-08-19T01:51:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-08-19T01:51:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-08-19T01:51:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-08-19T01:51:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-08-19T01:51:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-08-19T01:51:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-08-19T01:51:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-08-19T01:51:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 344, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1566701, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 344, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-08-19T01:51:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 344, 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1566701, 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:18.305 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:51:19.301 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-08-19T01:51:19.301 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-08-19T01:51:19.301 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-08-19T01:51:19.301 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-08-19T01:51:19.301 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-08-19T01:51:19.301 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-08-19T01:51:19.301 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-08-19T01:51:19.302 DEBUG:tasks.check_counter:Getting stats from j 2024-08-19T01:51:19.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.j perf dump 2024-08-19T01:51:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:19.984 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:19 smithi064 ceph-mon[18515]: pgmap v569: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 3.5 MiB/s wr, 136 op/s 2024-08-19T01:51:20.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:19 smithi123 ceph-mon[21829]: pgmap v569: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 3.5 MiB/s wr, 136 op/s 2024-08-19T01:51:20.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:19 smithi161 ceph-mon[21757]: pgmap v569: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 3.5 MiB/s wr, 136 op/s 2024-08-19T01:51:20.215 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-08-19T01:51:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 4, 2024-08-19T01:51:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 4, 2024-08-19T01:51:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 3072, 2024-08-19T01:51:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 448, 2024-08-19T01:51:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-08-19T01:51:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.003049928, 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000033316, 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.002187996, 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3, 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.003992548, 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001330849 2024-08-19T01:51:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 448 2024-08-19T01:51:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 348, 2024-08-19T01:51:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 337, 2024-08-19T01:51:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1617344, 2024-08-19T01:51:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 208256, 2024-08-19T01:51:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.052338022, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.020715181, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.045244463, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 336, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.014364277, 2024-08-19T01:51:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000042750 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1617344, 2024-08-19T01:51:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 208256 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 3, 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 259, 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 480, 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 280928, 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.019585158, 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.013790966, 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.002531919, 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 256, 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.018659493, 2024-08-19T01:51:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000072888 2024-08-19T01:51:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 480, 2024-08-19T01:51:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 280928 2024-08-19T01:51:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-08-19T01:51:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-08-19T01:51:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-08-19T01:51:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-08-19T01:51:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-08-19T01:51:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 148754, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 31, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-08-19T01:51:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-08-19T01:51:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 351, 2024-08-19T01:51:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1568248, 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 351, 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 351, 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1568248, 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-08-19T01:51:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-08-19T01:51:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-08-19T01:51:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-08-19T01:51:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-08-19T01:51:20.237 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-08-19T01:51:20.932 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-08-19T01:51:20.932 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-08-19T01:51:20.932 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-08-19T01:51:20.933 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-08-19T01:51:20.933 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-08-19T01:51:20.933 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-08-19T01:51:20.933 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-08-19T01:51:20.933 DEBUG:tasks.check_counter:Getting stats from b 2024-08-19T01:51:20.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.b perf dump 2024-08-19T01:51:21.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:20 smithi161 ceph-mon[21757]: pgmap v570: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-08-19T01:51:21.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:20 smithi064 ceph-mon[18515]: pgmap v570: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-08-19T01:51:21.237 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:51:21.359 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:20 smithi123 ceph-mon[21829]: pgmap v570: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-08-19T01:51:21.853 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-08-19T01:51:21.853 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:21.853 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 375836, 2024-08-19T01:51:21.853 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 248008, 2024-08-19T01:51:21.853 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 165976104, 2024-08-19T01:51:21.853 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 219627916, 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 16, 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 4, 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 16.678604162, 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 6.607597134, 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 12.098900580, 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.822854121, 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:21.854 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 248006, 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 13.310073742, 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000053668 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 581609, 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.089964081, 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000154 2024-08-19T01:51:21.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 601856, 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 184224 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 16635, 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 16684, 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 7434371, 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 94681640, 2024-08-19T01:51:21.856 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 77, 2024-08-19T01:51:21.857 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 9, 2024-08-19T01:51:21.857 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 1.844660898, 2024-08-19T01:51:21.857 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.480469924, 2024-08-19T01:51:21.857 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.385622631, 2024-08-19T01:51:21.857 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.897662621, 2024-08-19T01:51:21.857 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 16682, 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 3.405666535, 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000204152 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 364, 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000055844, 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000153 2024-08-19T01:51:21.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 1818912, 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 738432 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 266034, 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 365791, 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 85335123, 2024-08-19T01:51:21.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 298710478, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 15, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 6, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 20.141180315, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 9.481671465, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 13.313478542, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 1.006392780, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 365790, 2024-08-19T01:51:21.860 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 35.497822619, 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000097044 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 553814, 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.115323274, 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000208 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 63803168, 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 204700288 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-08-19T01:51:21.862 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 30224, 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 64.910959188, 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002147662 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-08-19T01:51:21.863 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 25199, 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 6.815295306, 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000270458 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 132495, 2024-08-19T01:51:21.864 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 97875, 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 97875, 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 1646.412755384, 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.016821586 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 34620, 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-08-19T01:51:21.865 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 164, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 17, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 9, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 4472, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 1431, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 2988, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 53, 2024-08-19T01:51:21.866 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 2423, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 365, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 466, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 466, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 13, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 183356, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 87742, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 34617, 2024-08-19T01:51:21.867 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 34, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 1074, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 0, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 16, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 353, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 0, 2024-08-19T01:51:21.868 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 0, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 744, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 221995625, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 6, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-08-19T01:51:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 12, 2024-08-19T01:51:21.870 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 13506, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 67439, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 37707, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 13573, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 4864, 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 62440 2024-08-19T01:51:21.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 236, 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 91, 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 57, 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 64, 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 554, 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 0, 2024-08-19T01:51:21.872 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 14767, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 14767, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-08-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_path": 16, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_inode": 3334, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 16, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 26, 2024-08-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 80806, 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-08-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 80806, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 126, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 126, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 8, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-08-19T01:51:21.876 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 4194304, 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 320951541, 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 0, 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 19150, 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 153.867788516, 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.008034871 2024-08-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 0 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 2524, 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 17534, 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 15010, 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 221, 2024-08-19T01:51:21.878 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 403, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 182, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 4472, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 34271, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 29799, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 466, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 17437, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 16971, 2024-08-19T01:51:21.879 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 90192, 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 231712 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 161330, 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 2996, 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 132495, 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 2181, 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 6794, 2024-08-19T01:51:21.880 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 15489, 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 432.656306741, 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.027933133 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2803, 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 63.009736229, 2024-08-19T01:51:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.022479392 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-08-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 34874, 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 298.627823851, 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.008563050 2024-08-19T01:51:21.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.884 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.885 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.886 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.886 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-08-19T01:51:21.886 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.887 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.887 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.887 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-08-19T01:51:21.887 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 18, 2024-08-19T01:51:21.887 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.007189499, 2024-08-19T01:51:21.887 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000399416 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 119, 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.811658926, 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.006820663 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-08-19T01:51:21.888 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 18, 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 10.824279651, 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.601348869 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 10, 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.027707309, 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002770730 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 25911, 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 309.611769467, 2024-08-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.011949047 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 3239, 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 144.020072916, 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.044464363 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-08-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1, 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001899180, 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001899180 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 12, 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 5.542044222, 2024-08-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.461837018 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-08-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 153, 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 71.969117238, 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.470386387 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 452, 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 2.822065213, 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.006243507 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.896 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 9, 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.021905981, 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002433997 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-08-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 14767, 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 306.459178961, 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.020752974 2024-08-19T01:51:21.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 2, 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 7, 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 5, 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 2, 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 28050, 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 14025, 2024-08-19T01:51:21.900 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 1065, 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:21.901 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:21.902 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:21.903 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 356455, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 1302, 2024-08-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 176, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 2, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:21.906 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 9800, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 198, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 10222671, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 94195, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:21.907 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 50730, 2024-08-19T01:51:21.908 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 322010004, 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 37, 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 50730, 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 50730, 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 449.052473813, 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.008851813 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-08-19T01:51:21.909 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 50730, 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 51305 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 50730, 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 366, 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 50364, 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-08-19T01:51:21.910 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 0, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 11, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 366, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 19516, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 232, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-08-19T01:51:21.911 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 17, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 30193, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 0, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 22, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-08-19T01:51:21.912 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-08-19T01:51:21.913 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-08-19T01:51:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-08-19T01:51:21.915 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-08-19T01:51:21.915 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-08-19T01:51:21.915 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-08-19T01:51:21.915 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-08-19T01:51:21.915 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-08-19T01:51:21.915 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 118, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 35, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 12, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 12, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 0, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-08-19T01:51:21.916 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 732, 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 0, 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 216 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 1, 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-08-19T01:51:21.917 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-08-19T01:51:21.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.918 INFO:teuthology.orchestra.run.smithi123.stdout: "osd-slow-ops": { 2024-08-19T01:51:21.918 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 3354 2024-08-19T01:51:21.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 16, 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 8, 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 29534, 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 14767, 2024-08-19T01:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 658501, 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 191376365, 2024-08-19T01:51:21.920 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 658501, 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 658501, 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 191376365, 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.921 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:21.922 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 50730, 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 323623071, 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 50730, 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 323623071, 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:21.923 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:21.924 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:21.925 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:21.925 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:21.925 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:21.925 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 50730, 2024-08-19T01:51:21.925 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 50730, 2024-08-19T01:51:21.926 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 50730, 2024-08-19T01:51:21.926 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 50730, 2024-08-19T01:51:21.926 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:21.926 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.926 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.926 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.927 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.927 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-08-19T01:51:21.927 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:21.927 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-08-19T01:51:21.927 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:21.928 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 14767, 2024-08-19T01:51:21.928 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 1613067, 2024-08-19T01:51:21.928 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:21.928 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 14767, 2024-08-19T01:51:21.928 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 366, 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 1613067, 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.929 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x564ad2710960": { 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 80806, 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 316757237, 2024-08-19T01:51:21.930 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 80806, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 19150, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 316757237, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:21.931 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:21.932 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.932 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:21.932 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-08-19T01:51:22.964 INFO:tasks.check_counter:Daemon mds.b mds.exported=16 2024-08-19T01:51:22.965 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-08-19T01:51:22.965 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=236 2024-08-19T01:51:22.965 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=91 2024-08-19T01:51:22.965 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-08-19T01:51:22.965 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=18 2024-08-19T01:51:22.965 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=12 2024-08-19T01:51:22.965 DEBUG:tasks.check_counter:Getting stats from e 2024-08-19T01:51:22.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.e perf dump 2024-08-19T01:51:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:51:23.351 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:23 smithi123 ceph-mon[21829]: pgmap v571: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 155 KiB/s wr, 28 op/s 2024-08-19T01:51:23.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:23 smithi161 ceph-mon[21757]: pgmap v571: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 155 KiB/s wr, 28 op/s 2024-08-19T01:51:23.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:23 smithi064 ceph-mon[18515]: pgmap v571: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 155 KiB/s wr, 28 op/s 2024-08-19T01:51:23.873 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-08-19T01:51:23.874 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:23.874 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 2378, 2024-08-19T01:51:23.874 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 2357, 2024-08-19T01:51:23.874 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 92661666, 2024-08-19T01:51:23.874 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 845264, 2024-08-19T01:51:23.874 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 7, 2024-08-19T01:51:23.874 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:23.875 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.266342043, 2024-08-19T01:51:23.875 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.078162723, 2024-08-19T01:51:23.875 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.140098520, 2024-08-19T01:51:23.875 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.048715918, 2024-08-19T01:51:23.875 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2356, 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.083994840, 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000035651 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.877 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 1694944, 2024-08-19T01:51:23.877 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 213712 2024-08-19T01:51:23.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.877 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:23.877 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 383, 2024-08-19T01:51:23.877 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 638, 2024-08-19T01:51:23.878 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 100762312, 2024-08-19T01:51:23.878 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 812490, 2024-08-19T01:51:23.878 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 12, 2024-08-19T01:51:23.878 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 5, 2024-08-19T01:51:23.878 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.099473198, 2024-08-19T01:51:23.878 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.016658287, 2024-08-19T01:51:23.879 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.068356622, 2024-08-19T01:51:23.879 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.009608945, 2024-08-19T01:51:23.879 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:23.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 635, 2024-08-19T01:51:23.879 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.037262696, 2024-08-19T01:51:23.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000058681 2024-08-19T01:51:23.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.880 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:23.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.880 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.880 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 480, 2024-08-19T01:51:23.881 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 691584 2024-08-19T01:51:23.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.881 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:23.881 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 1931, 2024-08-19T01:51:23.881 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 1931, 2024-08-19T01:51:23.881 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 126330038, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 613241, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 4, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.210265812, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.031290974, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.122130969, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.046362734, 2024-08-19T01:51:23.882 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1930, 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.058445090, 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000030282 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 448 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.884 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 6940, 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.149637443, 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000021561 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.885 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-08-19T01:51:23.886 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-08-19T01:51:23.886 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-08-19T01:51:23.886 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.886 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.886 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.886 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.887 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-08-19T01:51:23.887 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 0, 2024-08-19T01:51:23.887 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 0, 2024-08-19T01:51:23.887 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-08-19T01:51:23.887 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.887 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 0, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 0, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 0, 2024-08-19T01:51:23.888 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 0, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 17666, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 0, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 17666, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 0, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 17666, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 0, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 0, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 0, 2024-08-19T01:51:23.889 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 10, 2024-08-19T01:51:23.890 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 0, 2024-08-19T01:51:23.890 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 0, 2024-08-19T01:51:23.890 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 0, 2024-08-19T01:51:23.890 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 0, 2024-08-19T01:51:23.890 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-08-19T01:51:23.890 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 0, 2024-08-19T01:51:23.891 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 744, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 221995625, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 6, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-08-19T01:51:23.892 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 0, 2024-08-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 0, 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 0, 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 0, 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 0, 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 0 2024-08-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 0, 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 0, 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 0, 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 0, 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 0, 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 0, 2024-08-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-08-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_path": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_inode": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-08-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 0, 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 0, 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-08-19T01:51:23.898 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 80807, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 0, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 126, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 8, 2024-08-19T01:51:23.899 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 4194304, 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 321045246, 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 320951541, 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.900 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 80807 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 2495, 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 17505, 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 15010, 2024-08-19T01:51:23.901 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 159, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 276, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 117, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 17666, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 17998, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 332, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 0, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 0, 2024-08-19T01:51:23.902 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 0, 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 78844, 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 215328 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 0, 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 0, 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 0, 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 0, 2024-08-19T01:51:23.903 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 0, 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.904 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-08-19T01:51:23.905 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.907 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-08-19T01:51:23.907 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.907 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.907 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.907 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.908 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.909 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-08-19T01:51:23.909 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.909 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.909 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.909 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-08-19T01:51:23.910 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.910 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.910 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.910 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-08-19T01:51:23.910 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.911 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-08-19T01:51:23.912 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.912 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.912 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.912 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-08-19T01:51:23.914 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.914 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.914 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.914 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-08-19T01:51:23.914 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.916 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-08-19T01:51:23.916 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.916 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.916 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.916 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.918 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-08-19T01:51:23.918 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.918 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.918 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.918 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-08-19T01:51:23.919 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.919 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.919 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.919 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-08-19T01:51:23.919 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.920 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.920 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.920 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-08-19T01:51:23.920 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.921 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.921 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.921 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-08-19T01:51:23.921 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.921 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.922 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.922 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-08-19T01:51:23.922 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.922 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.922 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-08-19T01:51:23.923 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.924 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.924 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.924 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-08-19T01:51:23.924 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.924 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.924 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-08-19T01:51:23.925 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-08-19T01:51:23.926 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 2, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 7, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 5, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 5, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 0, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 0, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 3446, 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-08-19T01:51:23.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:23.928 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:23.929 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:23.930 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-08-19T01:51:23.931 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 351736, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 3113, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-08-19T01:51:23.932 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 9800, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 198, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:23.933 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 19118431, 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 141333, 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-08-19T01:51:23.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 4293, 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 26, 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 4, 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 4293, 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-08-19T01:51:23.935 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 4293, 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 5.436871889, 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001266450 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 4293, 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 4294 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.936 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 4293, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 4293, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 0, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 2674, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 0, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 1616, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 0, 2024-08-19T01:51:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-08-19T01:51:23.938 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 2, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 0, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-08-19T01:51:23.939 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-08-19T01:51:23.940 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-08-19T01:51:23.941 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 118, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 29, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 11, 2024-08-19T01:51:23.942 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 13, 2024-08-19T01:51:23.943 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 2, 2024-08-19T01:51:23.943 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-08-19T01:51:23.943 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 0, 2024-08-19T01:51:23.943 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 2, 2024-08-19T01:51:23.943 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 0 2024-08-19T01:51:23.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 0, 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: "osd-slow-ops": { 2024-08-19T01:51:23.944 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 0 2024-08-19T01:51:23.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.945 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-08-19T01:51:23.945 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-08-19T01:51:23.945 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 0, 2024-08-19T01:51:23.945 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-08-19T01:51:23.945 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 0, 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 0, 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 0, 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-08-19T01:51:23.946 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:23.947 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 4688, 2024-08-19T01:51:23.947 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 319309010, 2024-08-19T01:51:23.947 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:23.947 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 4688, 2024-08-19T01:51:23.947 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:23.947 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 4688, 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 319309010, 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.949 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-08-19T01:51:23.949 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:23.949 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-08-19T01:51:23.949 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:23.949 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:23.949 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:23.950 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:23.950 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:23.950 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 4293, 2024-08-19T01:51:23.950 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 316865094, 2024-08-19T01:51:23.950 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 4293, 2024-08-19T01:51:23.950 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 316865094, 2024-08-19T01:51:23.950 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:23.951 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.951 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.951 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.951 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-08-19T01:51:23.952 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:23.952 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-08-19T01:51:23.952 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:23.952 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:23.952 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:23.952 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:23.952 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:23.953 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 4293, 2024-08-19T01:51:23.953 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 4293, 2024-08-19T01:51:23.953 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 4293, 2024-08-19T01:51:23.953 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 4293, 2024-08-19T01:51:23.953 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:23.953 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.954 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.954 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.954 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-08-19T01:51:23.955 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:23.955 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-08-19T01:51:23.955 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:23.955 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:23.955 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:23.955 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.956 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x56227f308b40": { 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:23.957 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:23.958 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:23.959 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:23.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:23.959 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-08-19T01:51:24.948 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-08-19T01:51:24.948 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-08-19T01:51:24.948 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-08-19T01:51:24.948 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-08-19T01:51:24.948 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-08-19T01:51:24.948 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-08-19T01:51:24.948 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-08-19T01:51:24.949 DEBUG:tasks.check_counter:Getting stats from h 2024-08-19T01:51:24.949 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.h perf dump 2024-08-19T01:51:25.205 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:51:25.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:25 smithi123 ceph-mon[21829]: pgmap v572: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 151 KiB/s wr, 27 op/s 2024-08-19T01:51:25.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:25 smithi161 ceph-mon[21757]: pgmap v572: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 151 KiB/s wr, 27 op/s 2024-08-19T01:51:25.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:25 smithi064 ceph-mon[18515]: pgmap v572: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 151 KiB/s wr, 27 op/s 2024-08-19T01:51:25.770 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-08-19T01:51:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 13288, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 13327, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 4922763, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 71525608, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 17, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 6, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 1.338636276, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.472449757, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.267392741, 2024-08-19T01:51:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.542721000, 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 13324, 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 1.449070426, 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000108756 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 108, 2024-08-19T01:51:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000045999, 2024-08-19T01:51:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000425 2024-08-19T01:51:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 1714272, 2024-08-19T01:51:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 240608 2024-08-19T01:51:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 213625, 2024-08-19T01:51:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 234464, 2024-08-19T01:51:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 68888493, 2024-08-19T01:51:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 166858012, 2024-08-19T01:51:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 13, 2024-08-19T01:51:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 5, 2024-08-19T01:51:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 14.064283540, 2024-08-19T01:51:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 6.260072176, 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 9.424614973, 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.540491085, 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 234464, 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 14.358056011, 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000061237 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 410516, 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.069400154, 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000169 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 40781904, 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 112485152 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 337695, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 197843, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 156453829, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 88909488, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 13, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 6, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 13.264762223, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 5.090267952, 2024-08-19T01:51:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 10.374589578, 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.390276386, 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 197841, 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 6.109056522, 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000030878 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 513954, 2024-08-19T01:51:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.071587917, 2024-08-19T01:51:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000139 2024-08-19T01:51:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 320, 2024-08-19T01:51:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 568608 2024-08-19T01:51:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-08-19T01:51:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 23454, 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 31.701106988, 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001351629 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 15215, 2024-08-19T01:51:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 2.155648702, 2024-08-19T01:51:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000141679 2024-08-19T01:51:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-08-19T01:51:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 94996, 2024-08-19T01:51:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 57237, 2024-08-19T01:51:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 57237, 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 471.595767475, 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.008239351 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 37759, 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-08-19T01:51:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 29, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 2, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 0, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 1013, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 187, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 826, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 0, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 745, 2024-08-19T01:51:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 0, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 121, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 121, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 8, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 134775, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 51946, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 37757, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 105, 2024-08-19T01:51:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 680, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 0, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 0, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 0, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 4, 2024-08-19T01:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 176, 2024-08-19T01:51:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-08-19T01:51:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-08-19T01:51:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-08-19T01:51:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 654, 2024-08-19T01:51:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 211382635, 2024-08-19T01:51:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 6, 2024-08-19T01:51:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-08-19T01:51:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 5, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 2934, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 29152, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 16838, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 6999, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 2987, 2024-08-19T01:51:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 38311 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 66, 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 421, 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 261, 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 326, 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 121, 2024-08-19T01:51:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 1, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 8858, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 8857, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-08-19T01:51:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_path": 16, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_inode": 1536, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 0, 2024-08-19T01:51:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 2, 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 44328, 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-08-19T01:51:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-08-19T01:51:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 44328, 2024-08-19T01:51:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-08-19T01:51:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-08-19T01:51:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 67, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 67, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 5, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 4194304, 2024-08-19T01:51:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 153984357, 2024-08-19T01:51:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 0, 2024-08-19T01:51:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-08-19T01:51:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 12567, 2024-08-19T01:51:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 60.750525862, 2024-08-19T01:51:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.004834131 2024-08-19T01:51:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 0 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 851, 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 9708, 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 8857, 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 211, 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 290, 2024-08-19T01:51:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 79, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 1013, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 19339, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 18326, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 121, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 10134, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 10013, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 72604, 2024-08-19T01:51:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 223520 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 120881, 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 804, 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 94996, 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 1256, 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 3365, 2024-08-19T01:51:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 8887, 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 153.619593431, 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.017285877 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 984, 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 10.598645450, 2024-08-19T01:51:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.010770981 2024-08-19T01:51:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-08-19T01:51:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-08-19T01:51:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-08-19T01:51:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 20709, 2024-08-19T01:51:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 46.665581703, 2024-08-19T01:51:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002253396 2024-08-19T01:51:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-08-19T01:51:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-08-19T01:51:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-08-19T01:51:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 15710, 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 73.874959655, 2024-08-19T01:51:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.004702416 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1721, 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 63.190176438, 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.036717127 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-08-19T01:51:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-08-19T01:51:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 32, 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 17.573481768, 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.549171305 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 336, 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.416824828, 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001240550 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-08-19T01:51:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 8858, 2024-08-19T01:51:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 105.656504202, 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.011927805 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 2, 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 5, 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 3, 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 2, 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-08-19T01:51:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 17257, 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 8628, 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 1030, 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-08-19T01:51:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 327366, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 511, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 176, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 2, 2024-08-19T01:51:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-08-19T01:51:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-08-19T01:51:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 9800, 2024-08-19T01:51:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 198, 2024-08-19T01:51:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-08-19T01:51:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 3165563, 2024-08-19T01:51:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 25399, 2024-08-19T01:51:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-08-19T01:51:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 31274, 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 151969018, 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 33, 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 31274, 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 31274, 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 253.413064575, 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.008102994 2024-08-19T01:51:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 31274, 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 31640 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 31274, 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 279, 2024-08-19T01:51:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 30995, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 0, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 10, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 279, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 12846, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 150, 2024-08-19T01:51:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 2, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 17761, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-08-19T01:51:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 0, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 20, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-08-19T01:51:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-08-19T01:51:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-08-19T01:51:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-08-19T01:51:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-08-19T01:51:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-08-19T01:51:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-08-19T01:51:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 118, 2024-08-19T01:51:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-08-19T01:51:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 29, 2024-08-19T01:51:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 12, 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 17, 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 5, 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 408, 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 0, 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 164 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 1, 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "osd-slow-ops": { 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 1540 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 16, 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 8, 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 17714, 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 8857, 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-08-19T01:51:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 564604, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 174273175, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 564604, 2024-08-19T01:51:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 564604, 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 174273175, 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 31274, 2024-08-19T01:51:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 152935503, 2024-08-19T01:51:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 31274, 2024-08-19T01:51:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 152935503, 2024-08-19T01:51:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-08-19T01:51:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-08-19T01:51:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 31274, 2024-08-19T01:51:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 31274, 2024-08-19T01:51:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 31274, 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 31274, 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 8857, 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 966485, 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 8857, 2024-08-19T01:51:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 279, 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 966485, 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x55e126e76960": { 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 44328, 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 149790053, 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 44328, 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 12567, 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 149790053, 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:25.853 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-08-19T01:51:26.577 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-08-19T01:51:26.577 INFO:tasks.check_counter:Daemon mds.h mds.imported=4 2024-08-19T01:51:26.577 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=66 2024-08-19T01:51:26.577 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=421 2024-08-19T01:51:26.577 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-08-19T01:51:26.577 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-08-19T01:51:26.577 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-08-19T01:51:26.577 DEBUG:tasks.check_counter:Getting stats from k 2024-08-19T01:51:26.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.k perf dump 2024-08-19T01:51:26.831 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T01:51:27.226 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:27 smithi123 ceph-mon[21829]: pgmap v573: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 151 KiB/s wr, 27 op/s 2024-08-19T01:51:27.342 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 4, 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 4, 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 3072, 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 448, 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 4, 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 0, 2024-08-19T01:51:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.003057865, 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.000037468, 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.002306998, 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 3, 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.004131587, 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001377195 2024-08-19T01:51:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 448 2024-08-19T01:51:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 350, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 339, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 1617888, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 209536, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 2, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 1, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.049602461, 2024-08-19T01:51:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.018427508, 2024-08-19T01:51:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.044926001, 2024-08-19T01:51:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 338, 2024-08-19T01:51:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.015375455, 2024-08-19T01:51:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000045489 2024-08-19T01:51:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 1617888, 2024-08-19T01:51:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 209536 2024-08-19T01:51:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 3, 2024-08-19T01:51:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 261, 2024-08-19T01:51:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 480, 2024-08-19T01:51:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 281280, 2024-08-19T01:51:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.018862707, 2024-08-19T01:51:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.009884324, 2024-08-19T01:51:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.006139719, 2024-08-19T01:51:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 258, 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.023597101, 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000091461 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 480, 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 281280 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 0, 2024-08-19T01:51:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:27.354 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:27.355 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:27.356 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 148721, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 30, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:27.357 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 0, 2024-08-19T01:51:27.358 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 928, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 2, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-08-19T01:51:27.359 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 353, 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 1568508, 2024-08-19T01:51:27.360 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 353, 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 353, 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 1568508, 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-08-19T01:51:27.361 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-08-19T01:51:27.362 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-08-19T01:51:27.362 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:27.362 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-08-19T01:51:27.362 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-08-19T01:51:27.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:27 smithi161 ceph-mon[21757]: pgmap v573: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 151 KiB/s wr, 27 op/s 2024-08-19T01:51:27.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:27 smithi064 ceph-mon[18515]: pgmap v573: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 151 KiB/s wr, 27 op/s 2024-08-19T01:51:28.018 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-08-19T01:51:28.018 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-08-19T01:51:28.018 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-08-19T01:51:28.019 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-08-19T01:51:28.019 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-08-19T01:51:28.019 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-08-19T01:51:28.019 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-08-19T01:51:28.019 DEBUG:tasks.check_counter:Getting stats from c 2024-08-19T01:51:28.019 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.c perf dump 2024-08-19T01:51:28.329 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:51:28.877 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-08-19T01:51:28.878 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:28.878 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 3215, 2024-08-19T01:51:28.878 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 3214, 2024-08-19T01:51:28.878 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 54238774, 2024-08-19T01:51:28.878 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 1021235, 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 8, 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 4, 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.256566651, 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.069566123, 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.090062283, 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.079009927, 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:28.879 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 3213, 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.100711880, 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000031345 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.880 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 448 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 396, 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 651, 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 58825561, 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 730474, 2024-08-19T01:51:28.881 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 10, 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 7, 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.076980270, 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.015972677, 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.046781937, 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.009396292, 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 648, 2024-08-19T01:51:28.882 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.029474971, 2024-08-19T01:51:28.883 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000045486 2024-08-19T01:51:28.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.883 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:28.883 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.883 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.883 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 320, 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 605120 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 590, 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 570, 2024-08-19T01:51:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 39494374, 2024-08-19T01:51:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 277152, 2024-08-19T01:51:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 4, 2024-08-19T01:51:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.095322018, 2024-08-19T01:51:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.023879482, 2024-08-19T01:51:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.079189871, 2024-08-19T01:51:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.005285876, 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 569, 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.025605910, 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000045001 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1694784, 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 213552 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-08-19T01:51:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 6345, 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.139972757, 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000022060 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 0, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-08-19T01:51:28.891 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 9755, 2024-08-19T01:51:28.892 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 1, 2024-08-19T01:51:28.892 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 9754, 2024-08-19T01:51:28.892 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-08-19T01:51:28.892 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 9754, 2024-08-19T01:51:28.892 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-08-19T01:51:28.892 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-08-19T01:51:28.892 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 7, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-08-19T01:51:28.893 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-08-19T01:51:28.894 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-08-19T01:51:28.894 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-08-19T01:51:28.894 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-08-19T01:51:28.894 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-08-19T01:51:28.894 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-08-19T01:51:28.894 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-08-19T01:51:28.894 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-08-19T01:51:28.895 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-08-19T01:51:28.896 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-08-19T01:51:28.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.898 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-08-19T01:51:28.898 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-08-19T01:51:28.898 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-08-19T01:51:28.898 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-08-19T01:51:28.898 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 0, 2024-08-19T01:51:28.898 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-08-19T01:51:28.898 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-08-19T01:51:28.899 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_path": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_inode": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-08-19T01:51:28.900 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 0, 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-08-19T01:51:28.901 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 44328, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 0, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 67, 2024-08-19T01:51:28.902 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 5, 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 153984357, 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 153979467, 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.903 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 44328 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 766, 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 9623, 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 8857, 2024-08-19T01:51:28.904 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 59, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 61, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 2, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 9755, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 9755, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-08-19T01:51:28.905 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 63780, 2024-08-19T01:51:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 215328 2024-08-19T01:51:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-08-19T01:51:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-08-19T01:51:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-08-19T01:51:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-08-19T01:51:28.906 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-08-19T01:51:28.907 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.909 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.910 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-08-19T01:51:28.910 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.910 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.910 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.911 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.912 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-08-19T01:51:28.913 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.915 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-08-19T01:51:28.915 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.915 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.915 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.915 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-08-19T01:51:28.915 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.917 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.918 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-08-19T01:51:28.920 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.922 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-08-19T01:51:28.923 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.925 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 2, 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 5, 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 3, 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 4, 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-08-19T01:51:28.926 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 2956, 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:28.927 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:28.928 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:28.928 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:28.928 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:28.928 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:28.928 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:28.928 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:28.928 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:28.929 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:28.930 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 270924, 2024-08-19T01:51:28.931 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 949, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:28.932 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9800, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 198, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 8901136, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 59800, 2024-08-19T01:51:28.933 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 3804, 2024-08-19T01:51:28.934 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 21, 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 3, 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 3804, 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 3804, 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 5.930872338, 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.001559114 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-08-19T01:51:28.935 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 3804, 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 3805 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 3804, 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 3804, 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 0, 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-08-19T01:51:28.936 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 2381, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 0, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 1421, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 0, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 0, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-08-19T01:51:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 1, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-08-19T01:51:28.938 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-08-19T01:51:28.939 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-08-19T01:51:28.939 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-08-19T01:51:28.939 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-08-19T01:51:28.939 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-08-19T01:51:28.939 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-08-19T01:51:28.939 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-08-19T01:51:28.940 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-08-19T01:51:28.941 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 118, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 29, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 11, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 13, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 2, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-08-19T01:51:28.942 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 0, 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 2, 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 0 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 0, 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-08-19T01:51:28.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: "osd-slow-ops": { 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-08-19T01:51:28.944 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-08-19T01:51:28.945 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-08-19T01:51:28.945 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-08-19T01:51:28.945 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-08-19T01:51:28.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.945 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:28.945 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:28.945 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 4197, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 152157982, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 4197, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:28.946 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 4197, 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 152157982, 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-08-19T01:51:28.947 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:28.948 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-08-19T01:51:28.948 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:28.948 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:28.948 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:28.948 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 3804, 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 149793943, 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 3804, 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 149793943, 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.949 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.950 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.950 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.950 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-08-19T01:51:28.950 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 3804, 2024-08-19T01:51:28.951 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 3804, 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 3804, 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 3804, 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.952 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:28.953 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.954 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x55eab26a8b40": { 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:28.955 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:28.956 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:28.957 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.957 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:28.957 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-08-19T01:51:29.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:29 smithi161 ceph-mon[21757]: pgmap v574: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 60 KiB/s wr, 7 op/s 2024-08-19T01:51:29.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:29 smithi064 ceph-mon[18515]: pgmap v574: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 60 KiB/s wr, 7 op/s 2024-08-19T01:51:29.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:29 smithi123 ceph-mon[21829]: pgmap v574: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 60 KiB/s wr, 7 op/s 2024-08-19T01:51:29.975 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-08-19T01:51:29.975 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-08-19T01:51:29.975 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-08-19T01:51:29.976 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-08-19T01:51:29.976 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-08-19T01:51:29.976 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-08-19T01:51:29.976 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-08-19T01:51:29.976 DEBUG:tasks.check_counter:Getting stats from f 2024-08-19T01:51:29.976 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.f perf dump 2024-08-19T01:51:30.236 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:51:30.772 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-08-19T01:51:30.772 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:30.772 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1549, 2024-08-19T01:51:30.772 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1529, 2024-08-19T01:51:30.772 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 358523140, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 582442, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 7, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.309713040, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.042236045, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.251721527, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.023715899, 2024-08-19T01:51:30.773 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:30.774 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1528, 2024-08-19T01:51:30.774 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.046512331, 2024-08-19T01:51:30.774 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000030440 2024-08-19T01:51:30.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.774 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:30.774 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.774 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.775 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.775 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1695056, 2024-08-19T01:51:30.775 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 214192 2024-08-19T01:51:30.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.775 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:30.775 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1412, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1666, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 326304512, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 1054473, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 9, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 7, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.276580419, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.039697987, 2024-08-19T01:51:30.776 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.194616203, 2024-08-19T01:51:30.777 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.031596113, 2024-08-19T01:51:30.777 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:30.777 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1664, 2024-08-19T01:51:30.777 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.060956010, 2024-08-19T01:51:30.777 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000036632 2024-08-19T01:51:30.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.777 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 607360 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.778 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 3161, 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 3163, 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 491667432, 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 1005256, 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.489550021, 2024-08-19T01:51:30.779 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.070154514, 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.321430033, 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.076371491, 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 3161, 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.087124491, 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000027562 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.780 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:30.781 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.781 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.781 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.781 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-19T01:51:30.781 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 0 2024-08-19T01:51:30.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-08-19T01:51:30.782 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 8735, 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.248634136, 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000028464 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-08-19T01:51:30.783 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-08-19T01:51:30.784 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-08-19T01:51:30.785 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 0, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 44884, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 20522, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 24362, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 15567, 2024-08-19T01:51:30.786 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 5, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-08-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-08-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-08-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-08-19T01:51:30.790 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-08-19T01:51:30.791 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-08-19T01:51:30.791 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-08-19T01:51:30.791 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-19T01:51:30.791 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-19T01:51:30.791 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-08-19T01:51:30.791 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-08-19T01:51:30.791 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-08-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-08-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-08-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-08-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-08-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-08-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-08-19T01:51:30.793 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_path": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_inode": 0, 2024-08-19T01:51:30.794 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-08-19T01:51:30.795 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 203151, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-08-19T01:51:30.796 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 143, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 18, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 603011568, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 1177379492, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 1177367660, 2024-08-19T01:51:30.797 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 203151 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 1180, 2024-08-19T01:51:30.798 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 44787, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 43607, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 56, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 318, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 262, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 44884, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 44884, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-08-19T01:51:30.799 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 94948, 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 215328 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-08-19T01:51:30.800 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.801 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.802 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-08-19T01:51:30.803 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.805 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.806 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.807 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-08-19T01:51:30.808 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.810 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.811 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-08-19T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-08-19T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-08-19T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-08-19T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 2, 2024-08-19T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 6, 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 4, 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 5, 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 3424, 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-08-19T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-08-19T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-08-19T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 290515, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 1440, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9800, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 198, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 34769708, 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 211494, 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-08-19T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-08-19T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-08-19T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 5725, 2024-08-19T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 21, 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 3, 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 5725, 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 5725, 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 6.486859198, 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.001133075 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-08-19T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 5725, 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 5726 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 5725, 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 5725, 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 0, 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-08-19T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 3523, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 0, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 2200, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 0, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 0, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-08-19T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 0, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 1, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 0, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-08-19T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-08-19T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-08-19T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-08-19T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 118, 2024-08-19T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-08-19T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 29, 2024-08-19T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 11, 2024-08-19T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 11, 2024-08-19T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-08-19T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-08-19T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 0, 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 2, 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 0 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 0, 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-08-19T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "osd-slow-ops": { 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-08-19T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-08-19T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 6118, 2024-08-19T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1175921415, 2024-08-19T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 6118, 2024-08-19T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 6118, 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1175921415, 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-08-19T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-08-19T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 5725, 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 1173231482, 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 5725, 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1173231482, 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 5725, 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 5725, 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 5725, 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 5725, 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-08-19T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x55cab5444b40": { 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-08-19T01:51:31.378 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:31 smithi161 ceph-mon[21757]: pgmap v575: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 79 KiB/s wr, 11 op/s 2024-08-19T01:51:31.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:31 smithi064 ceph-mon[18515]: pgmap v575: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 79 KiB/s wr, 11 op/s 2024-08-19T01:51:31.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:31 smithi123 ceph-mon[21829]: pgmap v575: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 79 KiB/s wr, 11 op/s 2024-08-19T01:51:31.834 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-08-19T01:51:31.834 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-08-19T01:51:31.834 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-08-19T01:51:31.834 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-08-19T01:51:31.834 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-08-19T01:51:31.834 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-08-19T01:51:31.834 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-08-19T01:51:31.835 DEBUG:tasks.check_counter:Getting stats from i 2024-08-19T01:51:31.835 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.i perf dump 2024-08-19T01:51:32.110 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:51:32.622 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-08-19T01:51:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 744711, 2024-08-19T01:51:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1067581, 2024-08-19T01:51:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 229491085, 2024-08-19T01:51:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 1073535424, 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 10, 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 41.487220515, 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 18.495511212, 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 25.842271823, 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 2.007621188, 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1067582, 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 25.737109455, 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000024107 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1633854, 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.238379117, 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000145 2024-08-19T01:51:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 169313344, 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 595381344 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 63303, 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 63587, 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 15868738, 2024-08-19T01:51:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 599871454, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 17, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 8, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 6.415673530, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 3.080914306, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 1.096403449, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 1.843268826, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 63585, 2024-08-19T01:51:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 3.900322326, 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000061340 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 70, 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000013471, 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000192 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 6688, 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 620208 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 234071, 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 374152, 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 51488838, 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 471095065, 2024-08-19T01:51:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 11, 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 8, 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 14.346699383, 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 5.783438208, 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 7.180585997, 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 1.529433847, 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 374148, 2024-08-19T01:51:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 9.411792352, 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000025155 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 509593, 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.084428411, 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000165 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 7600, 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 10112 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-08-19T01:51:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 137729, 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 104.352206529, 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000757663 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-08-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 73692, 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 9.083834676, 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000123267 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 361050, 2024-08-19T01:51:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 345916, 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 345916, 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 844.189423694, 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.002440446 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 15134, 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-08-19T01:51:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 335, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 31, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 34, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 5057, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 3046, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 2011, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-08-19T01:51:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 695, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 64, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 64, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 8, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 527316, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 320088, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 15132, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 108, 2024-08-19T01:51:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 3828, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 12, 2024-08-19T01:51:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 177, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 654, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 211382635, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 6, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-08-19T01:51:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 3, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 15818, 2024-08-19T01:51:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 151415, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 79582, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 34497, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 16709, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 247048 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 78, 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 475, 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 289, 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 378, 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 93, 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-08-19T01:51:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 43607, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 43607, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-08-19T01:51:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_path": 16, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_inode": 1893, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 65, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-08-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 203154, 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 143664, 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 135191, 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 67963, 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-08-19T01:51:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 8473, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 340, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 212, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 197, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 143, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 6, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 15, 2024-08-19T01:51:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 603011568, 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 1177417148, 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 0, 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 68869, 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 140.589314893, 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.002041402 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 0 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 1274, 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 44881, 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 43607, 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 210, 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 570, 2024-08-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 360, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 5057, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 90836, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 85779, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 64, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 47282, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 47218, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 90136, 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 223520 2024-08-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 547632, 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 361050, 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 6031, 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 1239, 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-08-19T01:51:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 43600, 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 257.902988904, 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.005915206 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 3523, 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 9.314604212, 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.002643941 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 101173, 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 49.433350119, 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000488602 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2, 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000216674, 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000108337 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-08-19T01:51:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-08-19T01:51:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 143401, 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 265.606041598, 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.001852191 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 9191, 2024-08-19T01:51:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 76.100161023, 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.008279856 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-08-19T01:51:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 39, 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 53.670385778, 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 1.376163737 2024-08-19T01:51:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1380, 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.679081447, 2024-08-19T01:51:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000492088 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-08-19T01:51:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 43607, 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 131.482593939, 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.003015171 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-08-19T01:51:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 2, 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 6, 2024-08-19T01:51:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 4, 2024-08-19T01:51:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 2, 2024-08-19T01:51:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-08-19T01:51:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 12835, 2024-08-19T01:51:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 6417, 2024-08-19T01:51:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 1067, 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 366459, 2024-08-19T01:51:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 703, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 88, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 1, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9800, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 198, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 7314437, 2024-08-19T01:51:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 74779, 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-08-19T01:51:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 161087, 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 1185708168, 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 32, 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 161087, 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 161087, 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 588.674211407, 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.003654386 2024-08-19T01:51:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 161087, 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 162871 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 161087, 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 1281, 2024-08-19T01:51:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 159806, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 31, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 31, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 1281, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 70150, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 621, 2024-08-19T01:51:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 31, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 87880, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-08-19T01:51:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 62, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-08-19T01:51:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-08-19T01:51:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-08-19T01:51:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 118, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 29, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 12, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 12, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-08-19T01:51:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 2086, 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 0, 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 698 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 1, 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-08-19T01:51:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "osd-slow-ops": { 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 1839 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-08-19T01:51:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 16, 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 8, 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 87214, 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 43607, 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-08-19T01:51:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 1042081, 2024-08-19T01:51:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 182361168, 2024-08-19T01:51:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 1042081, 2024-08-19T01:51:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 1042081, 2024-08-19T01:51:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 182361168, 2024-08-19T01:51:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-08-19T01:51:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 161087, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 1190468419, 2024-08-19T01:51:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 161087, 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1190468419, 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 161087, 2024-08-19T01:51:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 161087, 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 161087, 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 161087, 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 43607, 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 4760251, 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 43607, 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 1281, 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 4760251, 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x5584572fc960": { 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 574, 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 203154, 2024-08-19T01:51:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1173222844, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 203154, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 68869, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1173222270, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:32.701 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-08-19T01:51:33.437 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:33 smithi161 ceph-mon[21757]: pgmap v576: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-08-19T01:51:33.468 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-08-19T01:51:33.469 INFO:tasks.check_counter:Daemon mds.i mds.imported=12 2024-08-19T01:51:33.469 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=78 2024-08-19T01:51:33.469 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=475 2024-08-19T01:51:33.469 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-08-19T01:51:33.469 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-08-19T01:51:33.469 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-08-19T01:51:33.469 DEBUG:tasks.check_counter:Getting stats from l 2024-08-19T01:51:33.470 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph daemon mds.l perf dump 2024-08-19T01:51:33.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:33 smithi064 ceph-mon[18515]: pgmap v576: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-08-19T01:51:33.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:33 smithi123 ceph-mon[21829]: pgmap v576: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-08-19T01:51:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T01:51:34.293 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-08-19T01:51:34.293 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-08-19T01:51:34.293 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 0, 2024-08-19T01:51:34.293 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 2, 2024-08-19T01:51:34.293 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 0, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 0, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 4, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 0, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.001031864, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.000000000, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.000179787, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 0, 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 0 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 353, 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 342, 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 1618704, 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 211456, 2024-08-19T01:51:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 2, 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 1, 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.051427974, 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.018992632, 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.047205794, 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 341, 2024-08-19T01:51:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.016473741, 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000048310 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1618704, 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 211456 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 7, 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 264, 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 3552, 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 281904, 2024-08-19T01:51:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 3, 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.016444955, 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.009886467, 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.003846918, 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 262, 2024-08-19T01:51:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.023044743, 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000087957 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-08-19T01:51:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 281904 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 0, 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-08-19T01:51:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-08-19T01:51:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-08-19T01:51:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-08-19T01:51:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-08-19T01:51:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 148754, 2024-08-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 31, 2024-08-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-08-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-08-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-08-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-08-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-08-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-08-19T01:51:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 928, 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 2, 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-08-19T01:51:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 356, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1568898, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 356, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-08-19T01:51:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 356, 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1568898, 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-08-19T01:51:34.311 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-08-19T01:51:34.448 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:34 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:51:34.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:34 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:51:34.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:34 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:51:34.449 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:34 smithi161 ceph-mon[21757]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:51:34.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:34 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:51:34.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:34 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:51:34.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:34 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:51:34.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:34 smithi064 ceph-mon[18515]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:51:34.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:34 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T01:51:34.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:34 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T01:51:34.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:34 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T01:51:34.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:34 smithi123 ceph-mon[21829]: from='mgr.15258 ' entity='mgr.x' 2024-08-19T01:51:34.968 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-08-19T01:51:34.968 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-08-19T01:51:34.968 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-08-19T01:51:34.968 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-08-19T01:51:34.968 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-08-19T01:51:34.968 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-08-19T01:51:34.968 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-08-19T01:51:34.969 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-08-19T01:51:34.979 INFO:tasks.kclient:Unmounting kernel clients... 2024-08-19T01:51:34.979 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:51:34.979 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-19T01:51:35.000 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-08-19T01:51:35.000 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-19T01:51:35.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd blocklist ls 2024-08-19T01:51:35.369 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:35.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.853+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/1024559486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 msgr2=0x7f1a58107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:35.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.853+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/1024559486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a58107c80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009930 tx=0x7f1a4802a420 comp rx=0 tx=0).stop 2024-08-19T01:51:35.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.853+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/1024559486 shutdown_connections 2024-08-19T01:51:35.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.853+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/1024559486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a58102240 0x7f1a5810e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:35.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.853+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/1024559486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a581018a0 0x7f1a58101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.853+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/1024559486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a58107c80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.853+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/1024559486 >> 172.21.15.64:0/1024559486 conn(0x7f1a580fd5e0 msgr2=0x7f1a580ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.854+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/1024559486 shutdown_connections 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.854+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/1024559486 wait complete. 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.854+0000 7f1a5d2ce640 1 Processor -- start 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.854+0000 7f1a5d2ce640 1 -- start start 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a5d2ce640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a581018a0 0x7f1a581981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a5d2ce640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a58102240 0x7f1a58198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a5d2ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a5819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a5d2ce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58111c60 con 0x7f1a581078a0 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a5d2ce640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58111ae0 con 0x7f1a58102240 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a5d2ce640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1a58111de0 con 0x7f1a581018a0 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a5819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a5819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43068/0 (socket says 172.21.15.64:43068) 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 -- 172.21.15.64:0/2561792950 learned_addr learned my addr 172.21.15.64:0/2561792950 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a56ffd640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a581018a0 0x7f1a581981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:35.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a567fc640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a58102240 0x7f1a58198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 -- 172.21.15.64:0/2561792950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a581018a0 msgr2=0x7f1a581981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a581018a0 0x7f1a581981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 -- 172.21.15.64:0/2561792950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a58102240 msgr2=0x7f1a58198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a58102240 0x7f1a58198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 -- 172.21.15.64:0/2561792950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5819d170 con 0x7f1a581078a0 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a577fe640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a5819ca90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000eb10 tx=0x7f1a5000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.855+0000 7f1a3ffff640 1 -- 172.21.15.64:0/2561792950 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50004b20 con 0x7f1a581078a0 2024-08-19T01:51:35.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.856+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a5819d460 con 0x7f1a581078a0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.856+0000 7f1a3ffff640 1 -- 172.21.15.64:0/2561792950 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a50004cc0 con 0x7f1a581078a0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.856+0000 7f1a3ffff640 1 -- 172.21.15.64:0/2561792950 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50010760 con 0x7f1a581078a0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.856+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a581a4d40 con 0x7f1a581078a0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.857+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a58101d00 con 0x7f1a581078a0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.859+0000 7f1a3ffff640 1 -- 172.21.15.64:0/2561792950 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1a50005000 con 0x7f1a581078a0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.859+0000 7f1a3ffff640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1a400a2cf0 0x7f1a400a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.859+0000 7f1a3ffff640 1 -- 172.21.15.64:0/2561792950 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9554+0+0 (secure 0 0 0) 0x7f1a500cff70 con 0x7f1a581078a0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.859+0000 7f1a56ffd640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1a400a2cf0 0x7f1a400a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.860+0000 7f1a56ffd640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1a400a2cf0 0x7f1a400a51b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009900 tx=0x7f1a480058d0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:35.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:35.863+0000 7f1a3ffff640 1 -- 172.21.15.64:0/2561792950 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f1a50053df0 con 0x7f1a581078a0 2024-08-19T01:51:36.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.050+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1a58109880 con 0x7f1a581078a0 2024-08-19T01:51:36.051 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:35 smithi064 ceph-mon[18515]: pgmap v577: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-08-19T01:51:36.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.050+0000 7f1a3ffff640 1 -- 172.21.15.64:0/2561792950 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v118) ==== 81+0+945 (secure 0 0 0) 0x7f1a5009c780 con 0x7f1a581078a0 2024-08-19T01:51:36.052 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/2663741075 2024-08-20T01:33:13.815234+0000 2024-08-19T01:51:36.052 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1810737759 2024-08-20T01:33:13.815234+0000 2024-08-19T01:51:36.052 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/3149805608 2024-08-20T01:24:10.525240+0000 2024-08-19T01:51:36.052 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3838744191 2024-08-20T01:24:33.597653+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/2403632526 2024-08-20T01:24:33.597653+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/3149805608 2024-08-20T01:24:10.525240+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/973985936 2024-08-20T01:24:10.525240+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/4090416923 2024-08-20T01:24:33.597653+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1532949280 2024-08-20T01:33:13.815234+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/2663741075 2024-08-20T01:33:13.815234+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1697666313 2024-08-20T01:33:13.815234+0000 2024-08-19T01:51:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1948699247 2024-08-20T01:24:10.525240+0000 2024-08-19T01:51:36.054 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1182511186 2024-08-20T01:24:10.525240+0000 2024-08-19T01:51:36.054 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/2403632526 2024-08-20T01:24:33.597653+0000 2024-08-19T01:51:36.054 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3202507182 2024-08-20T01:24:33.597653+0000 2024-08-19T01:51:36.054 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1603406100 2024-08-20T01:33:13.815234+0000 2024-08-19T01:51:36.055 INFO:teuthology.orchestra.run.smithi064.stderr:listed 16 entries 2024-08-19T01:51:36.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.052+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1a400a2cf0 msgr2=0x7f1a400a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:36.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.052+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1a400a2cf0 0x7f1a400a51b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009900 tx=0x7f1a480058d0 comp rx=0 tx=0).stop 2024-08-19T01:51:36.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.052+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 msgr2=0x7f1a5819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:36.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.052+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a5819ca90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000eb10 tx=0x7f1a5000c2d0 comp rx=0 tx=0).stop 2024-08-19T01:51:36.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 shutdown_connections 2024-08-19T01:51:36.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f1a400a2cf0 0x7f1a400a51b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:36.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a581078a0 0x7f1a5819ca90 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:36.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a58102240 0x7f1a58198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:36.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 --2- 172.21.15.64:0/2561792950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a581018a0 0x7f1a581981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:36.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 >> 172.21.15.64:0/2561792950 conn(0x7f1a580fd5e0 msgr2=0x7f1a580fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:36.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 shutdown_connections 2024-08-19T01:51:36.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:36.053+0000 7f1a5d2ce640 1 -- 172.21.15.64:0/2561792950 wait complete. 2024-08-19T01:51:36.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:35 smithi123 ceph-mon[21829]: pgmap v577: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-08-19T01:51:36.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:35 smithi161 ceph-mon[21757]: pgmap v577: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-08-19T01:51:36.950 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-19T01:51:36.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:51:36.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-19T01:51:37.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:36 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2561792950' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:51:37.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:36 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2561792950' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:51:37.200 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:36 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2561792950' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-08-19T01:51:37.202 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T01:51:37.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:51:37.203 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-19T01:51:37.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:51:37.219 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-08-19T01:51:37.274 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-19T01:51:37.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:51:37.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-19T01:51:37.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-19T01:51:37.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link delete ceph-brx 2024-08-19T01:51:37.440 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-08-19T01:51:37.451 INFO:tasks.cephadm:Teardown begin 2024-08-19T01:51:37.451 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-19T01:51:37.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:51:37.711 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:38.106 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:37 smithi064 ceph-mon[18515]: pgmap v578: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 129 KiB/s wr, 11 op/s 2024-08-19T01:51:38.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:37 smithi123 ceph-mon[21829]: pgmap v578: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 129 KiB/s wr, 11 op/s 2024-08-19T01:51:38.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:37 smithi161 ceph-mon[21757]: pgmap v578: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 129 KiB/s wr, 11 op/s 2024-08-19T01:51:38.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.187+0000 7ff38390b640 1 -- 172.21.15.64:0/2427346009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 msgr2=0x7ff37c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:38.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.187+0000 7ff38390b640 1 --2- 172.21.15.64:0/2427346009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c10e9a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff374009cd0 tx=0x7ff37402a8d0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 -- 172.21.15.64:0/2427346009 shutdown_connections 2024-08-19T01:51:38.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 --2- 172.21.15.64:0/2427346009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c10e9a0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 --2- 172.21.15.64:0/2427346009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c1034b0 0x7ff37c1078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 --2- 172.21.15.64:0/2427346009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff37c102b00 0x7ff37c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 -- 172.21.15.64:0/2427346009 >> 172.21.15.64:0/2427346009 conn(0x7ff37c0fc500 msgr2=0x7ff37c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:38.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 -- 172.21.15.64:0/2427346009 shutdown_connections 2024-08-19T01:51:38.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 -- 172.21.15.64:0/2427346009 wait complete. 2024-08-19T01:51:38.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 Processor -- start 2024-08-19T01:51:38.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 -- start start 2024-08-19T01:51:38.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c102b00 0x7ff37c100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:38.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff37c1034b0 0x7ff37c100ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:38.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.188+0000 7ff38390b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c1011e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:38.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff38390b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c111e00 con 0x7ff37c108030 2024-08-19T01:51:38.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff38390b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c111c80 con 0x7ff37c102b00 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff38390b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c111f80 con 0x7ff37c1034b0 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c1011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c1011e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43084/0 (socket says 172.21.15.64:43084) 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 -- 172.21.15.64:0/304962352 learned_addr learned my addr 172.21.15.64:0/304962352 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff380ea6640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff37c1034b0 0x7ff37c100ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff3816a7640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c102b00 0x7ff37c100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 -- 172.21.15.64:0/304962352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff37c1034b0 msgr2=0x7ff37c100ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff37c1034b0 0x7ff37c100ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 -- 172.21.15.64:0/304962352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c102b00 msgr2=0x7ff37c100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:38.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c102b00 0x7ff37c100760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 -- 172.21.15.64:0/304962352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c1a6840 con 0x7ff37c108030 2024-08-19T01:51:38.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff381ea8640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c1011e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff374009ca0 tx=0x7ff37402f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:38.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff3727fc640 1 -- 172.21.15.64:0/304962352 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374031860 con 0x7ff37c108030 2024-08-19T01:51:38.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff3727fc640 1 -- 172.21.15.64:0/304962352 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374003e50 con 0x7ff37c108030 2024-08-19T01:51:38.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37c1a6ad0 con 0x7ff37c108030 2024-08-19T01:51:38.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff3727fc640 1 -- 172.21.15.64:0/304962352 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740307f0 con 0x7ff37c108030 2024-08-19T01:51:38.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.189+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c1a7010 con 0x7ff37c108030 2024-08-19T01:51:38.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.190+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff37c102ee0 con 0x7ff37c108030 2024-08-19T01:51:38.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.192+0000 7ff3727fc640 1 -- 172.21.15.64:0/304962352 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff3740041d0 con 0x7ff37c108030 2024-08-19T01:51:38.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.192+0000 7ff3727fc640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3680a2cf0 0x7ff3680a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:38.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.192+0000 7ff3816a7640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3680a2cf0 0x7ff3680a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:38.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.193+0000 7ff3727fc640 1 -- 172.21.15.64:0/304962352 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9554+0+0 (secure 0 0 0) 0x7ff3740f0820 con 0x7ff37c108030 2024-08-19T01:51:38.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.193+0000 7ff3816a7640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3680a2cf0 0x7ff3680a51b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff378005fd0 tx=0x7ff37800b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:38.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.196+0000 7ff3727fc640 1 -- 172.21.15.64:0/304962352 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7ff3740bd140 con 0x7ff37c108030 2024-08-19T01:51:38.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.407+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff37c10a010 con 0x7ff37c108030 2024-08-19T01:51:38.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.408+0000 7ff3727fc640 1 -- 172.21.15.64:0/304962352 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11921 (secure 0 0 0) 0x7ff3740bca40 con 0x7ff37c108030 2024-08-19T01:51:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:51:38.409 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":32,"btime":"2024-08-19T01:49:07:146032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:49:06.572445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:51:38.410 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 32 2024-08-19T01:51:38.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.410+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3680a2cf0 msgr2=0x7ff3680a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:38.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.410+0000 7ff38390b640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3680a2cf0 0x7ff3680a51b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff378005fd0 tx=0x7ff37800b390 comp rx=0 tx=0).stop 2024-08-19T01:51:38.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.410+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 msgr2=0x7ff37c1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:38.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.410+0000 7ff38390b640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c1011e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff374009ca0 tx=0x7ff37402f420 comp rx=0 tx=0).stop 2024-08-19T01:51:38.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 shutdown_connections 2024-08-19T01:51:38.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7ff3680a2cf0 0x7ff3680a51b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff37c108030 0x7ff37c1011e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff37c1034b0 0x7ff37c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 --2- 172.21.15.64:0/304962352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c102b00 0x7ff37c100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:38.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 >> 172.21.15.64:0/304962352 conn(0x7ff37c0fc500 msgr2=0x7ff37c1060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:38.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 shutdown_connections 2024-08-19T01:51:38.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:38.411+0000 7ff38390b640 1 -- 172.21.15.64:0/304962352 wait complete. 2024-08-19T01:51:39.120 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:38 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/304962352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:39.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs dump --format=json 2024-08-19T01:51:39.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:38 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/304962352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:39.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:38 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/304962352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:39.412 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:39.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 -- 172.21.15.64:0/3285777375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 msgr2=0x7f192010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:39.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 --2- 172.21.15.64:0/3285777375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192010d770 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1918009900 tx=0x7f191802a480 comp rx=0 tx=0).stop 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 -- 172.21.15.64:0/3285777375 shutdown_connections 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 --2- 172.21.15.64:0/3285777375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192010d770 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 --2- 172.21.15.64:0/3285777375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19201022a0 0x7f19201026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 --2- 172.21.15.64:0/3285777375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1920100b50 0x7f1920100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 -- 172.21.15.64:0/3285777375 >> 172.21.15.64:0/3285777375 conn(0x7f19200fc500 msgr2=0x7f19200fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 -- 172.21.15.64:0/3285777375 shutdown_connections 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.857+0000 7f1926e16640 1 -- 172.21.15.64:0/3285777375 wait complete. 2024-08-19T01:51:39.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 Processor -- start 2024-08-19T01:51:39.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 -- start start 2024-08-19T01:51:39.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1920100b50 0x7f1920197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:39.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19201022a0 0x7f19201975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:39.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192019b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:39.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1920110cb0 con 0x7f1920102c20 2024-08-19T01:51:39.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1920110b30 con 0x7f19201022a0 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.858+0000 7f1926e16640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1920110e30 con 0x7f1920100b50 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192019b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192019b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41666/0 (socket says 172.21.15.64:41666) 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 -- 172.21.15.64:0/2363925336 learned_addr learned my addr 172.21.15.64:0/2363925336 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f191ffff640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19201022a0 0x7f19201975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f1924bb2640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1920100b50 0x7f1920197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 -- 172.21.15.64:0/2363925336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1920100b50 msgr2=0x7f1920197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:39.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1920100b50 0x7f1920197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 -- 172.21.15.64:0/2363925336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19201022a0 msgr2=0x7f19201975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19201022a0 0x7f19201975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 -- 172.21.15.64:0/2363925336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192019c040 con 0x7f1920102c20 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f191ffff640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19201022a0 0x7f19201975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f1924bb2640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1920100b50 0x7f1920197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f19253b3640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192019b960 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1918004770 tx=0x7f1918003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f191dffb640 1 -- 172.21.15.64:0/2363925336 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191803f070 con 0x7f1920102c20 2024-08-19T01:51:39.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192019c2d0 con 0x7f1920102c20 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f191dffb640 1 -- 172.21.15.64:0/2363925336 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1918031c30 con 0x7f1920102c20 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.859+0000 7f191dffb640 1 -- 172.21.15.64:0/2363925336 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918035030 con 0x7f1920102c20 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.860+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19201a3c10 con 0x7f1920102c20 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.861+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1920100f30 con 0x7f1920102c20 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.862+0000 7f191dffb640 1 -- 172.21.15.64:0/2363925336 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1918042070 con 0x7f1920102c20 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.863+0000 7f191dffb640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f190c0a2cf0 0x7f190c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.863+0000 7f1924bb2640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f190c0a2cf0 0x7f190c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.863+0000 7f191dffb640 1 -- 172.21.15.64:0/2363925336 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9554+0+0 (secure 0 0 0) 0x7f19180f0960 con 0x7f1920102c20 2024-08-19T01:51:39.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.863+0000 7f1924bb2640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f190c0a2cf0 0x7f190c0a51b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1910005fd0 tx=0x7f1910009520 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:39.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:39.866+0000 7f191dffb640 1 -- 172.21.15.64:0/2363925336 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f19180bd110 con 0x7f1920102c20 2024-08-19T01:51:40.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.120+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19201a3de0 con 0x7f1920102c20 2024-08-19T01:51:40.121 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:39 smithi064 ceph-mon[18515]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 107 KiB/s wr, 8 op/s 2024-08-19T01:51:40.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.121+0000 7f191dffb640 1 -- 172.21.15.64:0/2363925336 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11921 (secure 0 0 0) 0x7f19180bca10 con 0x7f1920102c20 2024-08-19T01:51:40.122 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:51:40.123 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":32,"btime":"2024-08-19T01:49:07:146032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1183738942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1183738942},{"type":"v1","addr":"172.21.15.64:6835","nonce":1183738942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/404621970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":404621970},{"type":"v1","addr":"172.21.15.64:6837","nonce":404621970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/2883191544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":2883191544},{"type":"v1","addr":"172.21.15.64:6839","nonce":2883191544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/799479773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":799479773},{"type":"v1","addr":"172.21.15.64:6841","nonce":799479773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/4151238815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":4151238815},{"type":"v1","addr":"172.21.15.161:6833","nonce":4151238815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/856914093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":856914093},{"type":"v1","addr":"172.21.15.123:6833","nonce":856914093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-08-19T01:30:52.282253+0000","modified":"2024-08-19T01:49:06.572445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24476,"mds_1":24452,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24415":{"gid":24415,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3530643584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3530643584},{"type":"v1","addr":"172.21.15.161:6835","nonce":3530643584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1326350329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1326350329},{"type":"v1","addr":"172.21.15.161:6837","nonce":1326350329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6835/2709386676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2709386676},{"type":"v1","addr":"172.21.15.123:6835","nonce":2709386676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2198702470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2198702470},{"type":"v1","addr":"172.21.15.161:6839","nonce":2198702470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/3425782983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3425782983},{"type":"v1","addr":"172.21.15.123:6837","nonce":3425782983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/2281975354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":2281975354},{"type":"v1","addr":"172.21.15.123:6839","nonce":2281975354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24476,24452,24436]},"id":1}]} 2024-08-19T01:51:40.123 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 32 2024-08-19T01:51:40.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f190c0a2cf0 msgr2=0x7f190c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:40.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f190c0a2cf0 0x7f190c0a51b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1910005fd0 tx=0x7f1910009520 comp rx=0 tx=0).stop 2024-08-19T01:51:40.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 msgr2=0x7f192019b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:40.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192019b960 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1918004770 tx=0x7f1918003c10 comp rx=0 tx=0).stop 2024-08-19T01:51:40.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 shutdown_connections 2024-08-19T01:51:40.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f190c0a2cf0 0x7f190c0a51b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:40.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1920102c20 0x7f192019b960 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:40.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19201022a0 0x7f19201975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:40.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 --2- 172.21.15.64:0/2363925336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1920100b50 0x7f1920197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:40.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 >> 172.21.15.64:0/2363925336 conn(0x7f19200fc500 msgr2=0x7f1920106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:40.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 shutdown_connections 2024-08-19T01:51:40.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:40.123+0000 7f1926e16640 1 -- 172.21.15.64:0/2363925336 wait complete. 2024-08-19T01:51:40.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:39 smithi123 ceph-mon[21829]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 107 KiB/s wr, 8 op/s 2024-08-19T01:51:40.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:39 smithi161 ceph-mon[21757]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 107 KiB/s wr, 8 op/s 2024-08-19T01:51:40.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T01:51:41.075 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:41.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:40 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2363925336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:41.197 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:40 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2363925336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:41.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:40 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2363925336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-08-19T01:51:41.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2480519165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c8074240 msgr2=0x7fb7c8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:41.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2480519165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c8074240 0x7fb7c8074620 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009ca0 tx=0x7fb7b802a950 comp rx=0 tx=0).stop 2024-08-19T01:51:41.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2480519165 shutdown_connections 2024-08-19T01:51:41.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2480519165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c810e180 0x7fb7c8110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2480519165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7c8074b60 0x7fb7c810dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2480519165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c8074240 0x7fb7c8074620 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2480519165 >> 172.21.15.64:0/2480519165 conn(0x7fb7c80fd620 msgr2=0x7fb7c80ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:41.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2480519165 shutdown_connections 2024-08-19T01:51:41.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2480519165 wait complete. 2024-08-19T01:51:41.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.594+0000 7fb7cdf69640 1 Processor -- start 2024-08-19T01:51:41.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7cdf69640 1 -- start start 2024-08-19T01:51:41.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7cdf69640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7c8074240 0x7fb7c819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:41.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7cdf69640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c8074b60 0x7fb7c819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:41.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7cdf69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c810e180 0x7fb7c819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7cdf69640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8113eb0 con 0x7fb7c810e180 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7cdf69640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8113d30 con 0x7fb7c8074240 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7cdf69640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8114030 con 0x7fb7c8074b60 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c810e180 0x7fb7c819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c810e180 0x7fb7c819ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41690/0 (socket says 172.21.15.64:41690) 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 -- 172.21.15.64:0/2844828639 learned_addr learned my addr 172.21.15.64:0/2844828639 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c77fe640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7c8074240 0x7fb7c819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:41.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c6ffd640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c8074b60 0x7fb7c819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:41.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 -- 172.21.15.64:0/2844828639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c8074b60 msgr2=0x7fb7c819a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:41.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c8074b60 0x7fb7c819a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 -- 172.21.15.64:0/2844828639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7c8074240 msgr2=0x7fb7c819a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:41.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7c8074240 0x7fb7c819a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 -- 172.21.15.64:0/2844828639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c819f3a0 con 0x7fb7c810e180 2024-08-19T01:51:41.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c6ffd640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c8074b60 0x7fb7c819a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.595+0000 7fb7c7fff640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c810e180 0x7fb7c819ecc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000d720 tx=0x7fb7c000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.596+0000 7fb7c4ff9640 1 -- 172.21.15.64:0/2844828639 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0017070 con 0x7fb7c810e180 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.596+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c819f690 con 0x7fb7c810e180 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.596+0000 7fb7c4ff9640 1 -- 172.21.15.64:0/2844828639 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7c000de30 con 0x7fb7c810e180 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.596+0000 7fb7c4ff9640 1 -- 172.21.15.64:0/2844828639 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0004d90 con 0x7fb7c810e180 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.596+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c81a6f70 con 0x7fb7c810e180 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.597+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c8074620 con 0x7fb7c810e180 2024-08-19T01:51:41.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.598+0000 7fb7c4ff9640 1 -- 172.21.15.64:0/2844828639 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb7c0005020 con 0x7fb7c810e180 2024-08-19T01:51:41.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.598+0000 7fb7c4ff9640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb7b40a2cf0 0x7fb7b40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:41.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.598+0000 7fb7c77fe640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb7b40a2cf0 0x7fb7b40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:41.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.598+0000 7fb7c4ff9640 1 -- 172.21.15.64:0/2844828639 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9554+0+0 (secure 0 0 0) 0x7fb7c00cf0c0 con 0x7fb7c810e180 2024-08-19T01:51:41.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.599+0000 7fb7c77fe640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb7b40a2cf0 0x7fb7b40a51b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8004ad0 tx=0x7fb7b8004a60 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:41.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.603+0000 7fb7c4ff9640 1 -- 172.21.15.64:0/2844828639 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fb7c009b8d0 con 0x7fb7c810e180 2024-08-19T01:51:41.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.779+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7c8109300 con 0x7fb7c810e180 2024-08-19T01:51:41.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.781+0000 7fb7c4ff9640 1 -- 172.21.15.64:0/2844828639 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+25396 (secure 0 0 0) 0x7fb7c009b1d0 con 0x7fb7c810e180 2024-08-19T01:51:41.782 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-08-19T01:51:41.782 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":118,"fsid":"88c1a048-5dc9-11ef-bccf-c7b262605968","created":"2024-08-19T01:23:48.740738+0000","modified":"2024-08-19T01:51:13.835412+0000","last_up_change":"2024-08-19T01:30:00.957989+0000","last_in_change":"2024-08-19T01:29:41.614944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T01:26:37.913262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-19T01:30:47.899062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-19T01:30:50.489331+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":117,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-08-19T01:30:56.822547+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":117,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8c3fb9c1-436e-47fd-ae54-4f88a7b757c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6803","nonce":3320376368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6805","nonce":3320376368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6809","nonce":3320376368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3320376368},{"type":"v1","addr":"172.21.15.64:6807","nonce":3320376368}]},"public_addr":"172.21.15.64:6803/3320376368","cluster_addr":"172.21.15.64:6805/3320376368","heartbeat_back_addr":"172.21.15.64:6809/3320376368","heartbeat_front_addr":"172.21.15.64:6807/3320376368","state":["exists","up"]},{"osd":1,"uuid":"7e1f4596-0fe3-4857-a061-1b173ea09ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6801","nonce":4102727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6803","nonce":4102727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6807","nonce":4102727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4102727954},{"type":"v1","addr":"172.21.15.123:6805","nonce":4102727954}]},"public_addr":"172.21.15.123:6801/4102727954","cluster_addr":"172.21.15.123:6803/4102727954","heartbeat_back_addr":"172.21.15.123:6807/4102727954","heartbeat_front_addr":"172.21.15.123:6805/4102727954","state":["exists","up"]},{"osd":2,"uuid":"b86cd6b3-a20f-4b12-82ca-c926be9cf348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6801","nonce":2169170995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6803","nonce":2169170995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6807","nonce":2169170995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":2169170995},{"type":"v1","addr":"172.21.15.161:6805","nonce":2169170995}]},"public_addr":"172.21.15.161:6801/2169170995","cluster_addr":"172.21.15.161:6803/2169170995","heartbeat_back_addr":"172.21.15.161:6807/2169170995","heartbeat_front_addr":"172.21.15.161:6805/2169170995","state":["exists","up"]},{"osd":3,"uuid":"c9312a92-2ed5-4111-8e98-1822e682fe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6811","nonce":18842482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6813","nonce":18842482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6817","nonce":18842482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":18842482},{"type":"v1","addr":"172.21.15.64:6815","nonce":18842482}]},"public_addr":"172.21.15.64:6811/18842482","cluster_addr":"172.21.15.64:6813/18842482","heartbeat_back_addr":"172.21.15.64:6817/18842482","heartbeat_front_addr":"172.21.15.64:6815/18842482","state":["exists","up"]},{"osd":4,"uuid":"eb75bf17-e4d5-4cea-9a6e-cbbbf5056f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6809","nonce":529776515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6811","nonce":529776515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6815","nonce":529776515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":529776515},{"type":"v1","addr":"172.21.15.123:6813","nonce":529776515}]},"public_addr":"172.21.15.123:6809/529776515","cluster_addr":"172.21.15.123:6811/529776515","heartbeat_back_addr":"172.21.15.123:6815/529776515","heartbeat_front_addr":"172.21.15.123:6813/529776515","state":["exists","up"]},{"osd":5,"uuid":"3566d78c-0daf-4391-bd31-1f8c95f93d35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6809","nonce":3335610027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6811","nonce":3335610027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6815","nonce":3335610027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3335610027},{"type":"v1","addr":"172.21.15.161:6813","nonce":3335610027}]},"public_addr":"172.21.15.161:6809/3335610027","cluster_addr":"172.21.15.161:6811/3335610027","heartbeat_back_addr":"172.21.15.161:6815/3335610027","heartbeat_front_addr":"172.21.15.161:6813/3335610027","state":["exists","up"]},{"osd":6,"uuid":"727df6d7-b958-434a-9812-a1b798a99b8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6819","nonce":3144814721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6821","nonce":3144814721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6825","nonce":3144814721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3144814721},{"type":"v1","addr":"172.21.15.64:6823","nonce":3144814721}]},"public_addr":"172.21.15.64:6819/3144814721","cluster_addr":"172.21.15.64:6821/3144814721","heartbeat_back_addr":"172.21.15.64:6825/3144814721","heartbeat_front_addr":"172.21.15.64:6823/3144814721","state":["exists","up"]},{"osd":7,"uuid":"30066775-0caa-4862-bd5b-45711ad1c571","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6817","nonce":2282165351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6819","nonce":2282165351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6823","nonce":2282165351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2282165351},{"type":"v1","addr":"172.21.15.123:6821","nonce":2282165351}]},"public_addr":"172.21.15.123:6817/2282165351","cluster_addr":"172.21.15.123:6819/2282165351","heartbeat_back_addr":"172.21.15.123:6823/2282165351","heartbeat_front_addr":"172.21.15.123:6821/2282165351","state":["exists","up"]},{"osd":8,"uuid":"c5b64329-e576-4c16-87ae-2180430f9ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6817","nonce":2639499139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6819","nonce":2639499139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6823","nonce":2639499139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":2639499139},{"type":"v1","addr":"172.21.15.161:6821","nonce":2639499139}]},"public_addr":"172.21.15.161:6817/2639499139","cluster_addr":"172.21.15.161:6819/2639499139","heartbeat_back_addr":"172.21.15.161:6823/2639499139","heartbeat_front_addr":"172.21.15.161:6821/2639499139","state":["exists","up"]},{"osd":9,"uuid":"50795cb0-f711-4f52-b8fa-e9583c165cae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6827","nonce":1073086660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6829","nonce":1073086660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6833","nonce":1073086660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":1073086660},{"type":"v1","addr":"172.21.15.64:6831","nonce":1073086660}]},"public_addr":"172.21.15.64:6827/1073086660","cluster_addr":"172.21.15.64:6829/1073086660","heartbeat_back_addr":"172.21.15.64:6833/1073086660","heartbeat_front_addr":"172.21.15.64:6831/1073086660","state":["exists","up"]},{"osd":10,"uuid":"fe117900-4a44-493a-b389-37c19ce9356a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6825","nonce":3299707057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6827","nonce":3299707057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6831","nonce":3299707057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3299707057},{"type":"v1","addr":"172.21.15.123:6829","nonce":3299707057}]},"public_addr":"172.21.15.123:6825/3299707057","cluster_addr":"172.21.15.123:6827/3299707057","heartbeat_back_addr":"172.21.15.123:6831/3299707057","heartbeat_front_addr":"172.21.15.123:6829/3299707057","state":["exists","up"]},{"osd":11,"uuid":"3d23aa4f-abcc-4207-aa16-d6e3975b613f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6825","nonce":564834779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6827","nonce":564834779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6831","nonce":564834779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":564834779},{"type":"v1","addr":"172.21.15.161:6829","nonce":564834779}]},"public_addr":"172.21.15.161:6825/564834779","cluster_addr":"172.21.15.161:6827/564834779","heartbeat_back_addr":"172.21.15.161:6831/564834779","heartbeat_front_addr":"172.21.15.161:6829/564834779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:25:52.299723+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:13.167442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:34.072006+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:26:55.695841+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:18.245919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:27:40.618671+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:03.231018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:25.951175+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:28:49.196726+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:12.598488+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:36.044111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T01:29:58.993748+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/2663741075":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1810737759":"2024-08-20T01:33:13.815234+0000","172.21.15.64:6800/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/3838744191":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6801/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:6801/3149805608":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/973985936":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/4090416923":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1532949280":"2024-08-20T01:33:13.815234+0000","172.21.15.64:6800/2663741075":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1697666313":"2024-08-20T01:33:13.815234+0000","172.21.15.64:0/1948699247":"2024-08-20T01:24:10.525240+0000","172.21.15.64:0/1182511186":"2024-08-20T01:24:10.525240+0000","172.21.15.64:6800/2403632526":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/3202507182":"2024-08-20T01:24:33.597653+0000","172.21.15.64:0/1603406100":"2024-08-20T01:33:13.815234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]},{"pool":4,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T01:51:41.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb7b40a2cf0 msgr2=0x7fb7b40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:41.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb7b40a2cf0 0x7fb7b40a51b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8004ad0 tx=0x7fb7b8004a60 comp rx=0 tx=0).stop 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c810e180 msgr2=0x7fb7c819ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c810e180 0x7fb7c819ecc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000d720 tx=0x7fb7c000dbe0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 shutdown_connections 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fb7b40a2cf0 0x7fb7b40a51b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7c810e180 0x7fb7c819ecc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c8074b60 0x7fb7c819a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 --2- 172.21.15.64:0/2844828639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7c8074240 0x7fb7c819a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:41.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.783+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 >> 172.21.15.64:0/2844828639 conn(0x7fb7c80fd620 msgr2=0x7fb7c80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:41.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.784+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 shutdown_connections 2024-08-19T01:51:41.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:41.784+0000 7fb7cdf69640 1 -- 172.21.15.64:0/2844828639 wait complete. 2024-08-19T01:51:42.121 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:41 smithi064 ceph-mon[18515]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 289 KiB/s wr, 12 op/s 2024-08-19T01:51:42.121 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:41 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/2844828639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:51:42.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:41 smithi123 ceph-mon[21829]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 289 KiB/s wr, 12 op/s 2024-08-19T01:51:42.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:41 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/2844828639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:51:42.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:41 smithi161 ceph-mon[21757]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 289 KiB/s wr, 12 op/s 2024-08-19T01:51:42.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:41 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/2844828639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T01:51:42.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs fail cephfs 2024-08-19T01:51:42.725 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:43.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 -- 172.21.15.64:0/1513426594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 msgr2=0x7f856c10e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:43.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 --2- 172.21.15.64:0/1513426594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c10e780 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8564009cd0 tx=0x7f856402a8e0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 -- 172.21.15.64:0/1513426594 shutdown_connections 2024-08-19T01:51:43.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 --2- 172.21.15.64:0/1513426594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c10e780 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 --2- 172.21.15.64:0/1513426594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f856c1018b0 0x7f856c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 --2- 172.21.15.64:0/1513426594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c1078b0 0x7f856c107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 -- 172.21.15.64:0/1513426594 >> 172.21.15.64:0/1513426594 conn(0x7f856c0fd5f0 msgr2=0x7f856c0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 -- 172.21.15.64:0/1513426594 shutdown_connections 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 -- 172.21.15.64:0/1513426594 wait complete. 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 Processor -- start 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.181+0000 7f857409e640 1 -- start start 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f857409e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c1018b0 0x7f856c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f857409e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f857409e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f856c1078b0 0x7f856c19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f857409e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f856c111cb0 con 0x7f856c102250 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f857409e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f856c111b30 con 0x7f856c1018b0 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f857409e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f856c111e30 con 0x7f856c1078b0 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41704/0 (socket says 172.21.15.64:41704) 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 -- 172.21.15.64:0/113273996 learned_addr learned my addr 172.21.15.64:0/113273996 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571e3a640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c1018b0 0x7f856c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f857263b640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f856c1078b0 0x7f856c19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 -- 172.21.15.64:0/113273996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f856c1078b0 msgr2=0x7f856c19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f856c1078b0 0x7f856c19ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 -- 172.21.15.64:0/113273996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c1018b0 msgr2=0x7f856c198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c1018b0 0x7f856c198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 -- 172.21.15.64:0/113273996 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856c19d140 con 0x7f856c102250 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.182+0000 7f8571639640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c1986d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f855c00d770 tx=0x7f855c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.183+0000 7f8562ffd640 1 -- 172.21.15.64:0/113273996 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c017070 con 0x7f856c102250 2024-08-19T01:51:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.183+0000 7f8562ffd640 1 -- 172.21.15.64:0/113273996 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f855c005500 con 0x7f856c102250 2024-08-19T01:51:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.183+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f856c19d430 con 0x7f856c102250 2024-08-19T01:51:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.183+0000 7f8562ffd640 1 -- 172.21.15.64:0/113273996 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c004e00 con 0x7f856c102250 2024-08-19T01:51:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.183+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f856c1a4d10 con 0x7f856c102250 2024-08-19T01:51:43.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.184+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f856c101d10 con 0x7f856c102250 2024-08-19T01:51:43.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.185+0000 7f8562ffd640 1 -- 172.21.15.64:0/113273996 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f855c004fa0 con 0x7f856c102250 2024-08-19T01:51:43.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.185+0000 7f8562ffd640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85580a2cf0 0x7f85580a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:43.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.186+0000 7f8571e3a640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85580a2cf0 0x7f85580a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:43.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.186+0000 7f8562ffd640 1 -- 172.21.15.64:0/113273996 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9554+0+0 (secure 0 0 0) 0x7f855c0cf2e0 con 0x7f856c102250 2024-08-19T01:51:43.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.186+0000 7f8571e3a640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85580a2cf0 0x7f85580a51b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f856800bb60 tx=0x7f856800b390 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:43.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.190+0000 7f8562ffd640 1 -- 172.21.15.64:0/113273996 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f855c09baf0 con 0x7f856c102250 2024-08-19T01:51:43.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.400+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f856c109e50 con 0x7f856c102250 2024-08-19T01:51:43.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.403+0000 7f8562ffd640 1 -- 172.21.15.64:0/113273996 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) ==== 161+0+0 (secure 0 0 0) 0x7f855c09b3f0 con 0x7f856c102250 2024-08-19T01:51:43.404 INFO:teuthology.orchestra.run.smithi064.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-19T01:51:43.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85580a2cf0 msgr2=0x7f85580a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:43.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85580a2cf0 0x7f85580a51b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f856800bb60 tx=0x7f856800b390 comp rx=0 tx=0).stop 2024-08-19T01:51:43.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 msgr2=0x7f856c1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:43.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c1986d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f855c00d770 tx=0x7f855c00dc30 comp rx=0 tx=0).stop 2024-08-19T01:51:43.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 shutdown_connections 2024-08-19T01:51:43.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f85580a2cf0 0x7f85580a51b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f856c1078b0 0x7f856c19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f856c102250 0x7f856c1986d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 --2- 172.21.15.64:0/113273996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c1018b0 0x7f856c198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:43.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.406+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 >> 172.21.15.64:0/113273996 conn(0x7f856c0fd5f0 msgr2=0x7f856c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:43.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.407+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 shutdown_connections 2024-08-19T01:51:43.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:43.407+0000 7f857409e640 1 -- 172.21.15.64:0/113273996 wait complete. 2024-08-19T01:51:43.699 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c[51957]: ignoring --setuser ceph since I am not root 2024-08-19T01:51:43.699 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c[51957]: ignoring --setgroup ceph since I am not root 2024-08-19T01:51:43.699 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mds[51963]: -- 172.21.15.161:0/1983080276 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b9dad69860 con 0x55b9dad46800 2024-08-19T01:51:43.699 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c[51957]: starting mds.c at 2024-08-19T01:51:43.700 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f[51646]: ignoring --setuser ceph since I am not root 2024-08-19T01:51:43.700 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f[51646]: ignoring --setgroup ceph since I am not root 2024-08-19T01:51:43.700 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mds[51650]: -- 172.21.15.161:0/454443221 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e78ac3860 con 0x558e78aa0800 2024-08-19T01:51:43.700 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f[51646]: starting mds.f at 2024-08-19T01:51:43.700 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i[52261]: ignoring --setuser ceph since I am not root 2024-08-19T01:51:43.700 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i[52261]: ignoring --setgroup ceph since I am not root 2024-08-19T01:51:43.701 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mds[52265]: -- 172.21.15.161:0/2924752614 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e28d97c20 con 0x555e28d75000 2024-08-19T01:51:43.701 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i[52261]: starting mds.i at 2024-08-19T01:51:43.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a[18511]: 2024-08-19T01:51:43.401+0000 7fe38bd4b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-19T01:51:43.819 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: ignoring --setuser ceph since I am not root 2024-08-19T01:51:43.819 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: ignoring --setgroup ceph since I am not root 2024-08-19T01:51:43.819 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mds[51739]: -- 172.21.15.123:0/1137726370 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa59f2fc20 con 0x55fa59f0c800 2024-08-19T01:51:43.819 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: starting mds.b at 2024-08-19T01:51:43.820 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e[51127]: ignoring --setuser ceph since I am not root 2024-08-19T01:51:43.820 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e[51127]: ignoring --setgroup ceph since I am not root 2024-08-19T01:51:43.820 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e[51127]: starting mds.e at 2024-08-19T01:51:43.820 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h[51430]: ignoring --setuser ceph since I am not root 2024-08-19T01:51:43.821 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h[51430]: ignoring --setgroup ceph since I am not root 2024-08-19T01:51:43.821 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mds[51434]: -- 172.21.15.123:0/1383837664 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623d7663860 con 0x5623d7641000 2024-08-19T01:51:43.821 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h[51430]: starting mds.h at 2024-08-19T01:51:44.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-mon[18515]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 270 KiB/s wr, 10 op/s 2024-08-19T01:51:44.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/113273996' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-08-19T01:51:44.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-mon[18515]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-08-19T01:51:44.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-mon[18515]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-19T01:51:44.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/113273996' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-08-19T01:51:44.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-mon[18515]: osdmap e119: 12 total, 12 up, 12 in 2024-08-19T01:51:44.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:43 smithi064 ceph-mon[18515]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-08-19T01:51:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mon[21829]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 270 KiB/s wr, 10 op/s 2024-08-19T01:51:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/113273996' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-08-19T01:51:44.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mon[21829]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-08-19T01:51:44.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mon[21829]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-19T01:51:44.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/113273996' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-08-19T01:51:44.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mon[21829]: osdmap e119: 12 total, 12 up, 12 in 2024-08-19T01:51:44.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:43 smithi123 ceph-mon[21829]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-08-19T01:51:44.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-08-19T01:51:44.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mon[21757]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 270 KiB/s wr, 10 op/s 2024-08-19T01:51:44.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/113273996' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-08-19T01:51:44.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mon[21757]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-08-19T01:51:44.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mon[21757]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-19T01:51:44.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/113273996' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-08-19T01:51:44.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mon[21757]: osdmap e119: 12 total, 12 up, 12 in 2024-08-19T01:51:44.200 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:43 smithi161 ceph-mon[21757]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-08-19T01:51:44.427 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:45.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 -- 172.21.15.64:0/2867688844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc08101150 msgr2=0x7fbc0810d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:45.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 --2- 172.21.15.64:0/2867688844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc08101150 0x7fbc0810d680 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009cd0 tx=0x7fbc0002a8e0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 -- 172.21.15.64:0/2867688844 shutdown_connections 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 --2- 172.21.15.64:0/2867688844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc08101150 0x7fbc0810d680 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 --2- 172.21.15.64:0/2867688844 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc081007b0 0x7fbc08100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 --2- 172.21.15.64:0/2867688844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc081067b0 0x7fbc08106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 -- 172.21.15.64:0/2867688844 >> 172.21.15.64:0/2867688844 conn(0x7fbc080fc4d0 msgr2=0x7fbc080fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.992+0000 7fbc1015c640 1 -- 172.21.15.64:0/2867688844 shutdown_connections 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 -- 172.21.15.64:0/2867688844 wait complete. 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 Processor -- start 2024-08-19T01:51:45.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 -- start start 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc081007b0 0x7fbc080fca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc08101150 0x7fbc080fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc081067b0 0x7fbc080fd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbc08110bd0 con 0x7fbc081007b0 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc08110a50 con 0x7fbc081067b0 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc1015c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc08110d50 con 0x7fbc08101150 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc0def8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc081007b0 0x7fbc080fca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:45.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc0def8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc081007b0 0x7fbc080fca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41720/0 (socket says 172.21.15.64:41720) 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.993+0000 7fbc0def8640 1 -- 172.21.15.64:0/3514325055 learned_addr learned my addr 172.21.15.64:0/3514325055 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0d6f7640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc08101150 0x7fbc080fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0e6f9640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc081067b0 0x7fbc080fd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0def8640 1 -- 172.21.15.64:0/3514325055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc08101150 msgr2=0x7fbc080fcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0def8640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc08101150 0x7fbc080fcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0def8640 1 -- 172.21.15.64:0/3514325055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc081067b0 msgr2=0x7fbc080fd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0def8640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc081067b0 0x7fbc080fd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0def8640 1 -- 172.21.15.64:0/3514325055 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc081a6660 con 0x7fbc081007b0 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0d6f7640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc08101150 0x7fbc080fcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0e6f9640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc081067b0 0x7fbc080fd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc0def8640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc081007b0 0x7fbc080fca70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d950 tx=0x7fbc0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbbfeffd640 1 -- 172.21.15.64:0/3514325055 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04017070 con 0x7fbc081007b0 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbbfeffd640 1 -- 172.21.15.64:0/3514325055 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc04009c30 con 0x7fbc081007b0 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbbfeffd640 1 -- 172.21.15.64:0/3514325055 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0400f6b0 con 0x7fbc081007b0 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc081a6950 con 0x7fbc081007b0 2024-08-19T01:51:45.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.994+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc081a6ee0 con 0x7fbc081007b0 2024-08-19T01:51:45.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.995+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc08100c10 con 0x7fbc081007b0 2024-08-19T01:51:45.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.996+0000 7fbbfeffd640 1 -- 172.21.15.64:0/3514325055 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbc040109d0 con 0x7fbc081007b0 2024-08-19T01:51:45.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.997+0000 7fbbfeffd640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fbbf40a2d70 0x7fbbf40a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:45.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.997+0000 7fbc0d6f7640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fbbf40a2d70 0x7fbbf40a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:45.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.997+0000 7fbbfeffd640 1 -- 172.21.15.64:0/3514325055 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9724+0+0 (secure 0 0 0) 0x7fbc040cf540 con 0x7fbc081007b0 2024-08-19T01:51:45.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:44.997+0000 7fbc0d6f7640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fbbf40a2d70 0x7fbbf40a5230 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8006fd0 tx=0x7fbbf8008040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:45.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.001+0000 7fbbfeffd640 1 -- 172.21.15.64:0/3514325055 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7fbc04009de0 con 0x7fbc081007b0 2024-08-19T01:51:45.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.216+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbc0802d5e0 con 0x7fbc081007b0 2024-08-19T01:51:45.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 323 KiB/s wr, 12 op/s 2024-08-19T01:51:45.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6834/4009352986,v1:172.21.15.161:6835/4009352986] up:boot 2024-08-19T01:51:45.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6834/3321768504,v1:172.21.15.123:6835/3321768504] up:boot 2024-08-19T01:51:45.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6836/3476302415,v1:172.21.15.161:6837/3476302415] up:boot 2024-08-19T01:51:45.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.161:6838/1425416193,v1:172.21.15.161:6839/1425416193] up:boot 2024-08-19T01:51:45.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6836/2749836837,v1:172.21.15.123:6837/2749836837] up:boot 2024-08-19T01:51:45.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: mds.? [v2:172.21.15.123:6838/462645012,v1:172.21.15.123:6839/462645012] up:boot 2024-08-19T01:51:45.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-08-19T01:51:45.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:51:45.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:51:45.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:51:45.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:51:45.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:51:45.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:51:45.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:45 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3514325055' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:51:45.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 323 KiB/s wr, 12 op/s 2024-08-19T01:51:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6834/4009352986,v1:172.21.15.161:6835/4009352986] up:boot 2024-08-19T01:51:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6834/3321768504,v1:172.21.15.123:6835/3321768504] up:boot 2024-08-19T01:51:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6836/3476302415,v1:172.21.15.161:6837/3476302415] up:boot 2024-08-19T01:51:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.161:6838/1425416193,v1:172.21.15.161:6839/1425416193] up:boot 2024-08-19T01:51:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6836/2749836837,v1:172.21.15.123:6837/2749836837] up:boot 2024-08-19T01:51:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: mds.? [v2:172.21.15.123:6838/462645012,v1:172.21.15.123:6839/462645012] up:boot 2024-08-19T01:51:45.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-08-19T01:51:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:51:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:51:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:51:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:51:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:51:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:51:45.667 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:45 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3514325055' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:51:45.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 323 KiB/s wr, 12 op/s 2024-08-19T01:51:45.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6834/4009352986,v1:172.21.15.161:6835/4009352986] up:boot 2024-08-19T01:51:45.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6834/3321768504,v1:172.21.15.123:6835/3321768504] up:boot 2024-08-19T01:51:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6836/3476302415,v1:172.21.15.161:6837/3476302415] up:boot 2024-08-19T01:51:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.161:6838/1425416193,v1:172.21.15.161:6839/1425416193] up:boot 2024-08-19T01:51:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6836/2749836837,v1:172.21.15.123:6837/2749836837] up:boot 2024-08-19T01:51:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: mds.? [v2:172.21.15.123:6838/462645012,v1:172.21.15.123:6839/462645012] up:boot 2024-08-19T01:51:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-08-19T01:51:45.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-08-19T01:51:45.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-08-19T01:51:45.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-08-19T01:51:45.701 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-08-19T01:51:45.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-08-19T01:51:45.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: from='mgr.15258 172.21.15.64:0/1200814749' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-08-19T01:51:45.702 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:45 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3514325055' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T01:51:45.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.827+0000 7fbbfeffd640 1 -- 172.21.15.64:0/3514325055 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) ==== 104+0+0 (secure 0 0 0) 0x7fbc0409bcb0 con 0x7fbc081007b0 2024-08-19T01:51:45.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fbbf40a2d70 msgr2=0x7fbbf40a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:45.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fbbf40a2d70 0x7fbbf40a5230 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8006fd0 tx=0x7fbbf8008040 comp rx=0 tx=0).stop 2024-08-19T01:51:45.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc081007b0 msgr2=0x7fbc080fca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:45.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc081007b0 0x7fbc080fca70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d950 tx=0x7fbc0400de10 comp rx=0 tx=0).stop 2024-08-19T01:51:45.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 shutdown_connections 2024-08-19T01:51:45.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7fbbf40a2d70 0x7fbbf40a5230 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc081067b0 0x7fbc080fd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc08101150 0x7fbc080fcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 --2- 172.21.15.64:0/3514325055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc081007b0 0x7fbc080fca70 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:45.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.829+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 >> 172.21.15.64:0/3514325055 conn(0x7fbc080fc4d0 msgr2=0x7fbc0806a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:45.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.830+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 shutdown_connections 2024-08-19T01:51:45.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:45.830+0000 7fbc1015c640 1 -- 172.21.15.64:0/3514325055 wait complete. 2024-08-19T01:51:46.465 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-19T01:51:46.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:46 smithi123 ceph-mon[21829]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-08-19T01:51:46.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:46 smithi123 ceph-mon[21829]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-19T01:51:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:46 smithi123 ceph-mon[21829]: Cluster is now healthy 2024-08-19T01:51:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:46 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3514325055' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-08-19T01:51:46.666 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:46 smithi123 ceph-mon[21829]: fsmap 12 up:standby 2024-08-19T01:51:46.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:46 smithi161 ceph-mon[21757]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-08-19T01:51:46.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:46 smithi161 ceph-mon[21757]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-19T01:51:46.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:46 smithi161 ceph-mon[21757]: Cluster is now healthy 2024-08-19T01:51:46.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:46 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3514325055' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-08-19T01:51:46.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:46 smithi161 ceph-mon[21757]: fsmap 12 up:standby 2024-08-19T01:51:46.729 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:46.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:46 smithi064 ceph-mon[18515]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-08-19T01:51:46.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:46 smithi064 ceph-mon[18515]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-19T01:51:46.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:46 smithi064 ceph-mon[18515]: Cluster is now healthy 2024-08-19T01:51:46.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:46 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3514325055' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-08-19T01:51:46.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:46 smithi064 ceph-mon[18515]: fsmap 12 up:standby 2024-08-19T01:51:47.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 -- 172.21.15.64:0/68607797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 msgr2=0x7f29ac10e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:47.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 --2- 172.21.15.64:0/68607797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac10e780 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009cd0 tx=0x7f29a402a8e0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 -- 172.21.15.64:0/68607797 shutdown_connections 2024-08-19T01:51:47.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 --2- 172.21.15.64:0/68607797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac10e780 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 --2- 172.21.15.64:0/68607797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29ac1018b0 0x7f29ac101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 --2- 172.21.15.64:0/68607797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ac1078b0 0x7f29ac107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 -- 172.21.15.64:0/68607797 >> 172.21.15.64:0/68607797 conn(0x7f29ac0fd5f0 msgr2=0x7f29ac0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:47.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 -- 172.21.15.64:0/68607797 shutdown_connections 2024-08-19T01:51:47.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.297+0000 7f29b440b640 1 -- 172.21.15.64:0/68607797 wait complete. 2024-08-19T01:51:47.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 Processor -- start 2024-08-19T01:51:47.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 -- start start 2024-08-19T01:51:47.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ac1018b0 0x7f29ac1982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:47.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:47.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29ac1078b0 0x7f29ac19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac111d50 con 0x7f29ac102250 2024-08-19T01:51:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac111bd0 con 0x7f29ac1018b0 2024-08-19T01:51:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b440b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac111ed0 con 0x7f29ac1078b0 2024-08-19T01:51:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41734/0 (socket says 172.21.15.64:41734) 2024-08-19T01:51:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 -- 172.21.15.64:0/3772760559 learned_addr learned my addr 172.21.15.64:0/3772760559 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:47.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b21a7640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ac1018b0 0x7f29ac1982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:47.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 -- 172.21.15.64:0/3772760559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29ac1078b0 msgr2=0x7f29ac19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:47.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29ac1078b0 0x7f29ac19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 -- 172.21.15.64:0/3772760559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ac1018b0 msgr2=0x7f29ac1982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:47.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ac1018b0 0x7f29ac1982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.298+0000 7f29b19a6640 1 -- 172.21.15.64:0/3772760559 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ac19d2a0 con 0x7f29ac102250 2024-08-19T01:51:47.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.299+0000 7f29b21a7640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ac1018b0 0x7f29ac1982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.299+0000 7f29b19a6640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac198830 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f299c00b4a0 tx=0x7f299c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.299+0000 7f29a37fe640 1 -- 172.21.15.64:0/3772760559 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c014070 con 0x7f29ac102250 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.299+0000 7f29a37fe640 1 -- 172.21.15.64:0/3772760559 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f299c010af0 con 0x7f29ac102250 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.299+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ac19d590 con 0x7f29ac102250 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.299+0000 7f29a37fe640 1 -- 172.21.15.64:0/3772760559 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c017d10 con 0x7f29ac102250 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.299+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ac1a4e70 con 0x7f29ac102250 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.300+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ac101d90 con 0x7f29ac102250 2024-08-19T01:51:47.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.301+0000 7f29a37fe640 1 -- 172.21.15.64:0/3772760559 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f299c017070 con 0x7f29ac102250 2024-08-19T01:51:47.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.301+0000 7f29a37fe640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29980a2d70 0x7f29980a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:47.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.302+0000 7f29b21a7640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29980a2d70 0x7f29980a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:47.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.302+0000 7f29a37fe640 1 -- 172.21.15.64:0/3772760559 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9724+0+0 (secure 0 0 0) 0x7f299c0ceb90 con 0x7f29ac102250 2024-08-19T01:51:47.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.302+0000 7f29b21a7640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29980a2d70 0x7f29980a5230 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009930 tx=0x7f29a8008040 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:47.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.306+0000 7f29a37fe640 1 -- 172.21.15.64:0/3772760559 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f299c09b410 con 0x7f29ac102250 2024-08-19T01:51:47.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.482+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f29ac02d5e0 con 0x7f29ac102250 2024-08-19T01:51:47.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.830+0000 7f29a37fe640 1 -- 172.21.15.64:0/3772760559 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) ==== 181+0+0 (secure 0 0 0) 0x7f299c09ad10 con 0x7f29ac102250 2024-08-19T01:51:47.831 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_metadata' removed 2024-08-19T01:51:47.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.834+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29980a2d70 msgr2=0x7f29980a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:47.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.834+0000 7f29b440b640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29980a2d70 0x7f29980a5230 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009930 tx=0x7f29a8008040 comp rx=0 tx=0).stop 2024-08-19T01:51:47.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.834+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 msgr2=0x7f29ac198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:47.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.834+0000 7f29b440b640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac198830 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f299c00b4a0 tx=0x7f299c00b960 comp rx=0 tx=0).stop 2024-08-19T01:51:47.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.835+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 shutdown_connections 2024-08-19T01:51:47.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.835+0000 7f29b440b640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f29980a2d70 0x7f29980a5230 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.835+0000 7f29b440b640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29ac1078b0 0x7f29ac19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.835+0000 7f29b440b640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29ac102250 0x7f29ac198830 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.835+0000 7f29b440b640 1 --2- 172.21.15.64:0/3772760559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29ac1018b0 0x7f29ac1982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:47.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.835+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 >> 172.21.15.64:0/3772760559 conn(0x7f29ac0fd5f0 msgr2=0x7f29ac0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:47.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.835+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 shutdown_connections 2024-08-19T01:51:47.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:47.837+0000 7f29b440b640 1 -- 172.21.15.64:0/3772760559 wait complete. 2024-08-19T01:51:47.877 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:47 smithi161 ceph-mon[21757]: pgmap v584: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 241 KiB/s wr, 9 op/s 2024-08-19T01:51:47.878 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:47 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3772760559' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:47 smithi123 ceph-mon[21829]: pgmap v584: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 241 KiB/s wr, 9 op/s 2024-08-19T01:51:48.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:47 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3772760559' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:48.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:47 smithi064 ceph-mon[18515]: pgmap v584: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 241 KiB/s wr, 9 op/s 2024-08-19T01:51:48.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:47 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3772760559' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:48.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-19T01:51:48.859 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:49.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:48 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3772760559' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:49.124 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:48 smithi064 ceph-mon[18515]: osdmap e120: 12 total, 12 up, 12 in 2024-08-19T01:51:49.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:48 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3772760559' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:49.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:48 smithi123 ceph-mon[21829]: osdmap e120: 12 total, 12 up, 12 in 2024-08-19T01:51:49.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:48 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3772760559' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:49.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:48 smithi161 ceph-mon[21757]: osdmap e120: 12 total, 12 up, 12 in 2024-08-19T01:51:49.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.352+0000 7f637d2c1640 1 -- 172.21.15.64:0/2638151812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6378102c20 msgr2=0x7f637810d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:49.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.352+0000 7f637d2c1640 1 --2- 172.21.15.64:0/2638151812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6378102c20 0x7f637810d770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6370009cd0 tx=0x7f637002a970 comp rx=0 tx=0).stop 2024-08-19T01:51:49.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.352+0000 7f637d2c1640 1 -- 172.21.15.64:0/2638151812 shutdown_connections 2024-08-19T01:51:49.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.352+0000 7f637d2c1640 1 --2- 172.21.15.64:0/2638151812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6378102c20 0x7f637810d770 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.352+0000 7f637d2c1640 1 --2- 172.21.15.64:0/2638151812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63781022a0 0x7f63781026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.352+0000 7f637d2c1640 1 --2- 172.21.15.64:0/2638151812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6378100b50 0x7f6378100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.352+0000 7f637d2c1640 1 -- 172.21.15.64:0/2638151812 >> 172.21.15.64:0/2638151812 conn(0x7f63780fc500 msgr2=0x7f63780fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.353+0000 7f637d2c1640 1 -- 172.21.15.64:0/2638151812 shutdown_connections 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.353+0000 7f637d2c1640 1 -- 172.21.15.64:0/2638151812 wait complete. 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.353+0000 7f637d2c1640 1 Processor -- start 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f637d2c1640 1 -- start start 2024-08-19T01:51:49.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f637d2c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6378100b50 0x7f637819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f637d2c1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63781022a0 0x7f637819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f637d2c1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6378102c20 0x7f637819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f637d2c1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6378110cb0 con 0x7f6378100b50 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f637d2c1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6378110b30 con 0x7f63781022a0 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f637d2c1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6378110e30 con 0x7f6378102c20 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f63777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6378102c20 0x7f637819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f6376ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6378100b50 0x7f637819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:49.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f63777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6378102c20 0x7f637819ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.64:41486/0 (socket says 172.21.15.64:41486) 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f63777fe640 1 -- 172.21.15.64:0/3423115109 learned_addr learned my addr 172.21.15.64:0/3423115109 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.354+0000 7f63767fc640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63781022a0 0x7f637819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f6376ffd640 1 -- 172.21.15.64:0/3423115109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6378102c20 msgr2=0x7f637819ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f6376ffd640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6378102c20 0x7f637819ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f6376ffd640 1 -- 172.21.15.64:0/3423115109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63781022a0 msgr2=0x7f637819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f6376ffd640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63781022a0 0x7f637819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f6376ffd640 1 -- 172.21.15.64:0/3423115109 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637819f390 con 0x7f6378100b50 2024-08-19T01:51:49.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f6376ffd640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6378100b50 0x7f637819a3e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f636800e9f0 tx=0x7f636800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f635ffff640 1 -- 172.21.15.64:0/3423115109 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636800ccc0 con 0x7f6378100b50 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637819f680 con 0x7f6378100b50 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63781a6f60 con 0x7f6378100b50 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f635ffff640 1 -- 172.21.15.64:0/3423115109 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6368005020 con 0x7f6378100b50 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.355+0000 7f635ffff640 1 -- 172.21.15.64:0/3423115109 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63680049e0 con 0x7f6378100b50 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.356+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6364005290 con 0x7f6378100b50 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.358+0000 7f635ffff640 1 -- 172.21.15.64:0/3423115109 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6368004cc0 con 0x7f6378100b50 2024-08-19T01:51:49.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.359+0000 7f635ffff640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f63600a2df0 0x7f63600a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:49.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.359+0000 7f635ffff640 1 -- 172.21.15.64:0/3423115109 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9170+0+0 (secure 0 0 0) 0x7f63680cf530 con 0x7f6378100b50 2024-08-19T01:51:49.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.359+0000 7f63767fc640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f63600a2df0 0x7f63600a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:49.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.360+0000 7f63767fc640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f63600a2df0 0x7f63600a52b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f636c0095f0 tx=0x7f636c005f90 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:49.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.365+0000 7f635ffff640 1 -- 172.21.15.64:0/3423115109 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f636809be60 con 0x7f6378100b50 2024-08-19T01:51:49.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.580+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6364005580 con 0x7f6378100b50 2024-08-19T01:51:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.845+0000 7f635ffff640 1 -- 172.21.15.64:0/3423115109 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) ==== 169+0+0 (secure 0 0 0) 0x7f636809b760 con 0x7f6378100b50 2024-08-19T01:51:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data' removed 2024-08-19T01:51:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f63600a2df0 msgr2=0x7f63600a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f63600a2df0 0x7f63600a52b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f636c0095f0 tx=0x7f636c005f90 comp rx=0 tx=0).stop 2024-08-19T01:51:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6378100b50 msgr2=0x7f637819a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6378100b50 0x7f637819a3e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f636800e9f0 tx=0x7f636800eeb0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 shutdown_connections 2024-08-19T01:51:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f63600a2df0 0x7f63600a52b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6378102c20 0x7f637819ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63781022a0 0x7f637819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 --2- 172.21.15.64:0/3423115109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6378100b50 0x7f637819a3e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 >> 172.21.15.64:0/3423115109 conn(0x7f63780fc500 msgr2=0x7f637806a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 shutdown_connections 2024-08-19T01:51:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:49.847+0000 7f637d2c1640 1 -- 172.21.15.64:0/3423115109 wait complete. 2024-08-19T01:51:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:49 smithi123 ceph-mon[21829]: pgmap v586: 97 pgs: 97 active+clean; 333 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:50.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:49 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3423115109' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:50.183 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:49 smithi064 ceph-mon[18515]: pgmap v586: 97 pgs: 97 active+clean; 333 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:50.183 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:49 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3423115109' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:49 smithi161 ceph-mon[21757]: pgmap v586: 97 pgs: 97 active+clean; 333 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:50.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:49 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3423115109' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:50.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-08-19T01:51:50.764 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T01:51:51.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:50 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3423115109' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:51.114 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:50 smithi064 ceph-mon[18515]: osdmap e121: 12 total, 12 up, 12 in 2024-08-19T01:51:51.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:50 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3423115109' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:51.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:50 smithi123 ceph-mon[21829]: osdmap e121: 12 total, 12 up, 12 in 2024-08-19T01:51:51.198 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:50 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3423115109' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:51.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:50 smithi161 ceph-mon[21757]: osdmap e121: 12 total, 12 up, 12 in 2024-08-19T01:51:51.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 -- 172.21.15.64:0/3602014439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 msgr2=0x7f010c107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:51.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 --2- 172.21.15.64:0/3602014439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c107c90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0108009930 tx=0x7f010802a420 comp rx=0 tx=0).stop 2024-08-19T01:51:51.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 -- 172.21.15.64:0/3602014439 shutdown_connections 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 --2- 172.21.15.64:0/3602014439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f010c102250 0x7f010c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 --2- 172.21.15.64:0/3602014439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f010c1018b0 0x7f010c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 --2- 172.21.15.64:0/3602014439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c107c90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 -- 172.21.15.64:0/3602014439 >> 172.21.15.64:0/3602014439 conn(0x7f010c0fd5f0 msgr2=0x7f010c0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 -- 172.21.15.64:0/3602014439 shutdown_connections 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.339+0000 7f011411a640 1 -- 172.21.15.64:0/3602014439 wait complete. 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 Processor -- start 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 -- start start 2024-08-19T01:51:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f010c1018b0 0x7f010c074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f010c102250 0x7f010c074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c074ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f010c111cc0 con 0x7f010c1078b0 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f010c111b40 con 0x7f010c102250 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f011411a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f010c111e40 con 0x7f010c1018b0 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c074ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c074ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47510/0 (socket says 172.21.15.64:47510) 2024-08-19T01:51:51.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 -- 172.21.15.64:0/3097787265 learned_addr learned my addr 172.21.15.64:0/3097787265 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01116b5640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f010c102250 0x7f010c074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f0111eb6640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f010c1018b0 0x7f010c074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 -- 172.21.15.64:0/3097787265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f010c1018b0 msgr2=0x7f010c074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f010c1018b0 0x7f010c074220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 -- 172.21.15.64:0/3097787265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f010c102250 msgr2=0x7f010c074760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f010c102250 0x7f010c074760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 -- 172.21.15.64:0/3097787265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c1a65e0 con 0x7f010c1078b0 2024-08-19T01:51:51.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.340+0000 7f01126b7640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c074ca0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f010400eb10 tx=0x7f010400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.341+0000 7f0102ffd640 1 -- 172.21.15.64:0/3097787265 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010400ce20 con 0x7f010c1078b0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.341+0000 7f0102ffd640 1 -- 172.21.15.64:0/3097787265 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0104005020 con 0x7f010c1078b0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.341+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010c1a68d0 con 0x7f010c1078b0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.341+0000 7f0102ffd640 1 -- 172.21.15.64:0/3097787265 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104010690 con 0x7f010c1078b0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.341+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f010c1a6dc0 con 0x7f010c1078b0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.342+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c101d10 con 0x7f010c1078b0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.343+0000 7f0102ffd640 1 -- 172.21.15.64:0/3097787265 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0104005540 con 0x7f010c1078b0 2024-08-19T01:51:51.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.343+0000 7f0102ffd640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f00f80a2cf0 0x7f00f80a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-19T01:51:51.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.343+0000 7f0111eb6640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f00f80a2cf0 0x7f00f80a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-19T01:51:51.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.343+0000 7f0102ffd640 1 -- 172.21.15.64:0/3097787265 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8582+0+0 (secure 0 0 0) 0x7f01040cf9b0 con 0x7f010c1078b0 2024-08-19T01:51:51.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.344+0000 7f0111eb6640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f00f80a2cf0 0x7f00f80a51b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0108006010 tx=0x7f0108000bc0 comp rx=0 tx=0).ready entity=mgr.15258 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-19T01:51:51.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.348+0000 7f0102ffd640 1 -- 172.21.15.64:0/3097787265 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187871 (secure 0 0 0) 0x7f010409c590 con 0x7f010c1078b0 2024-08-19T01:51:51.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.526+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f010c1a7080 con 0x7f010c1078b0 2024-08-19T01:51:51.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.868+0000 7f0102ffd640 1 -- 172.21.15.64:0/3097787265 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v122) ==== 178+0+0 (secure 0 0 0) 0x7f010409be90 con 0x7f010c1078b0 2024-08-19T01:51:51.870 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data_ec' removed 2024-08-19T01:51:51.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f00f80a2cf0 msgr2=0x7f00f80a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:51.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f00f80a2cf0 0x7f00f80a51b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0108006010 tx=0x7f0108000bc0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 msgr2=0x7f010c074ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T01:51:51.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c074ca0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f010400eb10 tx=0x7f010400ef10 comp rx=0 tx=0).stop 2024-08-19T01:51:51.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 shutdown_connections 2024-08-19T01:51:51.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:6800/3917059587,v1:172.21.15.64:6801/3917059587] conn(0x7f00f80a2cf0 0x7f00f80a51b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f010c1078b0 0x7f010c074ca0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f010c102250 0x7f010c074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.871+0000 7f011411a640 1 --2- 172.21.15.64:0/3097787265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f010c1018b0 0x7f010c074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-19T01:51:51.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.872+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 >> 172.21.15.64:0/3097787265 conn(0x7f010c0fd5f0 msgr2=0x7f010c0fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T01:51:51.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.873+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 shutdown_connections 2024-08-19T01:51:51.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-08-19T01:51:51.873+0000 7f011411a640 1 -- 172.21.15.64:0/3097787265 wait complete. 2024-08-19T01:51:52.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:51 smithi123 ceph-mon[21829]: pgmap v588: 33 pgs: 33 active+clean; 333 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:52.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:51 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3097787265' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:51 smithi161 ceph-mon[21757]: pgmap v588: 33 pgs: 33 active+clean; 333 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:52.199 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:51 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3097787265' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:52.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:51 smithi064 ceph-mon[18515]: pgmap v588: 33 pgs: 33 active+clean; 333 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:52.234 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:51 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3097787265' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T01:51:52.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:51:52.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:51:52.663 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:51:52.719 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-08-19T01:51:52.720 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-19T01:51:52.735 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-19T01:51:52.750 DEBUG:teuthology.orchestra.run.smithi161:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-19T01:51:52.768 INFO:tasks.cephadm:Stopping all daemons... 2024-08-19T01:51:52.769 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-08-19T01:51:52.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a 2024-08-19T01:51:52.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:52 smithi123 ceph-mon[21829]: from='client.? 172.21.15.64:0/3097787265' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:52.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:52 smithi123 ceph-mon[21829]: osdmap e122: 12 total, 12 up, 12 in 2024-08-19T01:51:52.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:52 smithi123 ceph-mon[21829]: pgmap v590: 1 pgs: 1 active+clean; 577 KiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:52.948 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:52 smithi161 ceph-mon[21757]: from='client.? 172.21.15.64:0/3097787265' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:52.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:52 smithi161 ceph-mon[21757]: osdmap e122: 12 total, 12 up, 12 in 2024-08-19T01:51:52.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:52 smithi161 ceph-mon[21757]: pgmap v590: 1 pgs: 1 active+clean; 577 KiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:53.073 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:52 smithi064 systemd[1]: Stopping Ceph mon.a for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:51:53.073 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:52 smithi064 ceph-mon[18515]: from='client.? 172.21.15.64:0/3097787265' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T01:51:53.074 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:52 smithi064 ceph-mon[18515]: osdmap e122: 12 total, 12 up, 12 in 2024-08-19T01:51:53.074 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:52 smithi064 ceph-mon[18515]: pgmap v590: 1 pgs: 1 active+clean; 577 KiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T01:51:53.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a[18511]: 2024-08-19T01:51:53.071+0000 7fe391556640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:51:53.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a[18511]: 2024-08-19T01:51:53.071+0000 7fe391556640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-08-19T01:51:53.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:53 smithi064 podman[86532]: 2024-08-19 01:51:53.123303604 +0000 UTC m=+0.147578723 container died de0cedb2b02e2950d14257a9b359cdde999dce0a18f4703586f571d9f6f2d43d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-19T01:51:53.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:53 smithi064 podman[86532]: 2024-08-19 01:51:53.506067551 +0000 UTC m=+0.530342678 container cleanup de0cedb2b02e2950d14257a9b359cdde999dce0a18f4703586f571d9f6f2d43d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-19T01:51:53.734 INFO:journalctl@ceph.mon.a.smithi064.stdout:Aug 19 01:51:53 smithi064 bash[86532]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-a 2024-08-19T01:51:53.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.a.service' 2024-08-19T01:51:54.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:51:54.014 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-08-19T01:51:54.014 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-08-19T01:51:54.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.b 2024-08-19T01:51:54.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:54 smithi123 systemd[1]: Stopping Ceph mon.b for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:51:54.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-b[21825]: 2024-08-19T01:51:54.391+0000 7f7f76d41640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:51:54.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-b[21825]: 2024-08-19T01:51:54.391+0000 7f7f76d41640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-08-19T01:51:54.665 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:54 smithi123 podman[58885]: 2024-08-19 01:51:54.525206529 +0000 UTC m=+0.263201095 container died b7d3a7dc2949b637a6aafae0b5b7c4605b5adfa9d5a1a720b6c4ff355fd177a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-08-19T01:51:55.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:54 smithi123 podman[58885]: 2024-08-19 01:51:54.926237742 +0000 UTC m=+0.664232299 container cleanup b7d3a7dc2949b637a6aafae0b5b7c4605b5adfa9d5a1a720b6c4ff355fd177a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-b, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:51:55.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:54 smithi123 bash[58885]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-b 2024-08-19T01:51:55.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.b.service' 2024-08-19T01:51:55.438 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:55 smithi123 podman[58900]: 2024-08-19 01:51:55.186315673 +0000 UTC m=+0.659454320 container remove b7d3a7dc2949b637a6aafae0b5b7c4605b5adfa9d5a1a720b6c4ff355fd177a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-08-19T01:51:55.438 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:55 smithi123 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.b.service: Deactivated successfully. 2024-08-19T01:51:55.438 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:55 smithi123 systemd[1]: Stopped Ceph mon.b for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:51:55.439 INFO:journalctl@ceph.mon.b.smithi123.stdout:Aug 19 01:51:55 smithi123 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.b.service: Consumed 15.092s CPU time. 2024-08-19T01:51:55.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:51:55.463 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-08-19T01:51:55.463 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-08-19T01:51:55.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.c 2024-08-19T01:51:55.949 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:55 smithi161 systemd[1]: Stopping Ceph mon.c for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:51:56.367 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-c[21752]: 2024-08-19T01:51:55.949+0000 7fcd41af2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:51:56.367 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-c[21752]: 2024-08-19T01:51:55.949+0000 7fcd41af2640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-08-19T01:51:56.367 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:56 smithi161 podman[59972]: 2024-08-19 01:51:56.082386998 +0000 UTC m=+0.297187877 container died 425a47cf752d9afa6b4f2e095f21918f7afacf5943aedfd11c91eafc630373fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-19T01:51:56.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:56 smithi161 podman[59972]: 2024-08-19 01:51:56.466645695 +0000 UTC m=+0.681446574 container cleanup 425a47cf752d9afa6b4f2e095f21918f7afacf5943aedfd11c91eafc630373fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:51:56.699 INFO:journalctl@ceph.mon.c.smithi161.stdout:Aug 19 01:51:56 smithi161 bash[59972]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mon-c 2024-08-19T01:51:56.878 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mon.c.service' 2024-08-19T01:51:56.930 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:51:56.930 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-08-19T01:51:56.930 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-08-19T01:51:56.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.x 2024-08-19T01:51:57.234 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:51:56 smithi064 systemd[1]: Stopping Ceph mgr.x for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:51:57.588 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:51:57 smithi064 podman[86655]: 2024-08-19 01:51:57.291930105 +0000 UTC m=+0.146405624 container died ce16acf65597807653cd6526ec776381e234811f503f7420782d6ecda484279b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-08-19T01:51:57.984 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:51:57 smithi064 podman[86655]: 2024-08-19 01:51:57.787079252 +0000 UTC m=+0.641554780 container cleanup ce16acf65597807653cd6526ec776381e234811f503f7420782d6ecda484279b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-19T01:51:57.984 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Aug 19 01:51:57 smithi064 bash[86655]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-x 2024-08-19T01:51:58.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.x.service' 2024-08-19T01:51:58.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:51:58.269 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-08-19T01:51:58.269 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-08-19T01:51:58.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.y 2024-08-19T01:51:58.637 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:51:58 smithi123 systemd[1]: Stopping Ceph mgr.y for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:51:58.915 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:51:58 smithi123 podman[59006]: 2024-08-19 01:51:58.636177588 +0000 UTC m=+0.154714819 container died 6c99f9ff5c165213516e6f9df5703104f9eeb3aa4fde080bb8185d389f7f53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-08-19T01:51:59.665 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:51:59 smithi123 podman[59006]: 2024-08-19 01:51:59.274386549 +0000 UTC m=+0.792923769 container cleanup 6c99f9ff5c165213516e6f9df5703104f9eeb3aa4fde080bb8185d389f7f53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, ceph=True) 2024-08-19T01:51:59.665 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:51:59 smithi123 bash[59006]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y 2024-08-19T01:51:59.665 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:51:59 smithi123 podman[59020]: 2024-08-19 01:51:59.496479354 +0000 UTC m=+0.840709490 container remove 6c99f9ff5c165213516e6f9df5703104f9eeb3aa4fde080bb8185d389f7f53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-y, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:51:59.666 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Aug 19 01:51:59 smithi123 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-08-19T01:51:59.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.y.service' 2024-08-19T01:51:59.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:51:59.802 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-08-19T01:51:59.802 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-08-19T01:51:59.802 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.z 2024-08-19T01:52:00.159 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:51:59 smithi161 systemd[1]: Stopping Ceph mgr.z for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:00.448 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:00 smithi161 podman[60099]: 2024-08-19 01:52:00.158943997 +0000 UTC m=+0.154101101 container died feace770023ab161a854ce28dcc016e0c0ce14129698c9c14bf4a8d67e06424b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:52:00.824 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:00 smithi161 podman[60099]: 2024-08-19 01:52:00.572298081 +0000 UTC m=+0.567455178 container cleanup feace770023ab161a854ce28dcc016e0c0ce14129698c9c14bf4a8d67e06424b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-08-19T01:52:00.824 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:00 smithi161 bash[60099]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z 2024-08-19T01:52:01.069 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.z.service' 2024-08-19T01:52:01.091 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:00 smithi161 podman[60111]: 2024-08-19 01:52:00.823723342 +0000 UTC m=+0.657226274 container remove feace770023ab161a854ce28dcc016e0c0ce14129698c9c14bf4a8d67e06424b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-19T01:52:01.091 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:00 smithi161 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-08-19T01:52:01.091 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:01 smithi161 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.z.service: Failed with result 'exit-code'. 2024-08-19T01:52:01.091 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:01 smithi161 systemd[1]: Stopped Ceph mgr.z for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:52:01.092 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Aug 19 01:52:01 smithi161 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mgr.z.service: Consumed 11.893s CPU time. 2024-08-19T01:52:01.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:01.105 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-08-19T01:52:01.105 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-08-19T01:52:01.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.0 2024-08-19T01:52:01.418 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:01 smithi064 systemd[1]: Stopping Ceph osd.0 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:01.734 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0[29576]: 2024-08-19T01:52:01.416+0000 7f747013b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:01.734 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0[29576]: 2024-08-19T01:52:01.416+0000 7f747013b640 -1 osd.0 122 *** Got signal Terminated *** 2024-08-19T01:52:01.735 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0[29576]: 2024-08-19T01:52:01.416+0000 7f747013b640 -1 osd.0 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:07.153 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:06 smithi064 podman[86776]: 2024-08-19 01:52:06.875214812 +0000 UTC m=+5.572576633 container died f9d4456113155dc446b79d1913f5e208a915c8e5afc9e762fe4177306872bb54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-08-19T01:52:07.484 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:07 smithi064 podman[86776]: 2024-08-19 01:52:07.342989419 +0000 UTC m=+6.040351269 container cleanup f9d4456113155dc446b79d1913f5e208a915c8e5afc9e762fe4177306872bb54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T01:52:07.484 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:07 smithi064 bash[86776]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0 2024-08-19T01:52:08.234 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:07 smithi064 podman[86792]: 2024-08-19 01:52:07.978366281 +0000 UTC m=+1.094167411 container remove f9d4456113155dc446b79d1913f5e208a915c8e5afc9e762fe4177306872bb54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-08-19T01:52:08.672 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:08 smithi064 podman[86884]: 2024-08-19 01:52:08.238878861 +0000 UTC m=+0.017858175 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:52:08.672 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:08 smithi064 podman[86884]: 2024-08-19 01:52:08.371060948 +0000 UTC m=+0.150040295 container create 8deb5444ac319486e0301ba26f367a3484f6cdd3662b987e4450fc048f8d98a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-08-19T01:52:08.949 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:08 smithi064 podman[86884]: 2024-08-19 01:52:08.671355577 +0000 UTC m=+0.450334879 container init 8deb5444ac319486e0301ba26f367a3484f6cdd3662b987e4450fc048f8d98a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-08-19T01:52:08.950 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:08 smithi064 podman[86884]: 2024-08-19 01:52:08.675622203 +0000 UTC m=+0.454601507 container start 8deb5444ac319486e0301ba26f367a3484f6cdd3662b987e4450fc048f8d98a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=) 2024-08-19T01:52:08.950 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:08 smithi064 podman[86884]: 2024-08-19 01:52:08.949196972 +0000 UTC m=+0.728176356 container attach 8deb5444ac319486e0301ba26f367a3484f6cdd3662b987e4450fc048f8d98a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-08-19T01:52:08.950 INFO:journalctl@ceph.osd.0.smithi064.stdout:Aug 19 01:52:08 smithi064 podman[86884]: 2024-08-19 01:52:08.949489285 +0000 UTC m=+0.728468604 container died 8deb5444ac319486e0301ba26f367a3484f6cdd3662b987e4450fc048f8d98a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-0-deactivate, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T01:52:09.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.0.service' 2024-08-19T01:52:09.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:09.713 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-08-19T01:52:09.713 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-19T01:52:09.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.1 2024-08-19T01:52:10.038 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:09 smithi123 systemd[1]: Stopping Ceph osd.1 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:10.415 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1[26267]: 2024-08-19T01:52:10.037+0000 7f267af38640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:10.415 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1[26267]: 2024-08-19T01:52:10.037+0000 7f267af38640 -1 osd.1 122 *** Got signal Terminated *** 2024-08-19T01:52:10.415 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1[26267]: 2024-08-19T01:52:10.037+0000 7f267af38640 -1 osd.1 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:15.906 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:15 smithi123 podman[59132]: 2024-08-19 01:52:15.593950466 +0000 UTC m=+5.676040079 container died 6ef4f404cfc1aaafdf1605e0b5c19c6d7fd7ea9f75c059b99c1ccbf46f6e3dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS) 2024-08-19T01:52:16.165 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:15 smithi123 podman[59132]: 2024-08-19 01:52:15.995651677 +0000 UTC m=+6.077741290 container cleanup 6ef4f404cfc1aaafdf1605e0b5c19c6d7fd7ea9f75c059b99c1ccbf46f6e3dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-08-19T01:52:16.165 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:15 smithi123 bash[59132]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1 2024-08-19T01:52:16.613 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:16 smithi123 podman[59149]: 2024-08-19 01:52:16.289044198 +0000 UTC m=+0.691338820 container remove 6ef4f404cfc1aaafdf1605e0b5c19c6d7fd7ea9f75c059b99c1ccbf46f6e3dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-19T01:52:16.916 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:16 smithi123 podman[59238]: 2024-08-19 01:52:16.515437122 +0000 UTC m=+0.017280688 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:52:16.916 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:16 smithi123 podman[59238]: 2024-08-19 01:52:16.648353742 +0000 UTC m=+0.150197341 container create 4d2c511f609e694623cc1c1af7bdc4bcaab89080476a287f3c500477db79dc45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-08-19T01:52:17.166 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:16 smithi123 podman[59238]: 2024-08-19 01:52:16.918240543 +0000 UTC m=+0.420084110 container init 4d2c511f609e694623cc1c1af7bdc4bcaab89080476a287f3c500477db79dc45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-19T01:52:17.166 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:16 smithi123 podman[59238]: 2024-08-19 01:52:16.921899803 +0000 UTC m=+0.423743366 container start 4d2c511f609e694623cc1c1af7bdc4bcaab89080476a287f3c500477db79dc45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0) 2024-08-19T01:52:17.166 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:17 smithi123 podman[59238]: 2024-08-19 01:52:17.026475664 +0000 UTC m=+0.528319221 container attach 4d2c511f609e694623cc1c1af7bdc4bcaab89080476a287f3c500477db79dc45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-08-19T01:52:17.167 INFO:journalctl@ceph.osd.1.smithi123.stdout:Aug 19 01:52:17 smithi123 podman[59238]: 2024-08-19 01:52:17.165623301 +0000 UTC m=+0.667466870 container died 4d2c511f609e694623cc1c1af7bdc4bcaab89080476a287f3c500477db79dc45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-1-deactivate, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-19T01:52:17.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.1.service' 2024-08-19T01:52:17.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:17.904 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-19T01:52:17.904 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-19T01:52:17.904 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.2 2024-08-19T01:52:18.198 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:17 smithi161 systemd[1]: Stopping Ceph osd.2 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:18.461 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2[26704]: 2024-08-19T01:52:18.409+0000 7fcca6516640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:18.462 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2[26704]: 2024-08-19T01:52:18.409+0000 7fcca6516640 -1 osd.2 122 *** Got signal Terminated *** 2024-08-19T01:52:18.462 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2[26704]: 2024-08-19T01:52:18.409+0000 7fcca6516640 -1 osd.2 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:24.151 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:23 smithi161 podman[60222]: 2024-08-19 01:52:23.847985355 +0000 UTC m=+5.737996514 container died d3b317e57230e8890acbdea34c74a388fbd46f73b77c433c75263592a0202714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2, RELEASE=squid-de09dd0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:52:24.448 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:24 smithi161 podman[60222]: 2024-08-19 01:52:24.234133022 +0000 UTC m=+6.124144183 container cleanup d3b317e57230e8890acbdea34c74a388fbd46f73b77c433c75263592a0202714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-19T01:52:24.449 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:24 smithi161 bash[60222]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2 2024-08-19T01:52:24.825 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:24 smithi161 podman[60251]: 2024-08-19 01:52:24.502434441 +0000 UTC m=+0.650659925 container remove d3b317e57230e8890acbdea34c74a388fbd46f73b77c433c75263592a0202714 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-08-19T01:52:25.131 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:24 smithi161 podman[60337]: 2024-08-19 01:52:24.72732887 +0000 UTC m=+0.017038931 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:52:25.131 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:24 smithi161 podman[60337]: 2024-08-19 01:52:24.861873978 +0000 UTC m=+0.151584024 container create 99462702f1bfdfc914aa198dccdbb9195d155274e221d8b4d7c39e711ac89bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-08-19T01:52:25.131 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:25 smithi161 podman[60337]: 2024-08-19 01:52:25.131091946 +0000 UTC m=+0.420802004 container init 99462702f1bfdfc914aa198dccdbb9195d155274e221d8b4d7c39e711ac89bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-08-19T01:52:25.448 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:25 smithi161 podman[60337]: 2024-08-19 01:52:25.134549142 +0000 UTC m=+0.424259195 container start 99462702f1bfdfc914aa198dccdbb9195d155274e221d8b4d7c39e711ac89bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, ceph=True) 2024-08-19T01:52:25.449 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:25 smithi161 podman[60337]: 2024-08-19 01:52:25.239499843 +0000 UTC m=+0.529209905 container attach 99462702f1bfdfc914aa198dccdbb9195d155274e221d8b4d7c39e711ac89bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD) 2024-08-19T01:52:25.449 INFO:journalctl@ceph.osd.2.smithi161.stdout:Aug 19 01:52:25 smithi161 podman[60337]: 2024-08-19 01:52:25.366887973 +0000 UTC m=+0.656598019 container died 99462702f1bfdfc914aa198dccdbb9195d155274e221d8b4d7c39e711ac89bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-2-deactivate, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T01:52:26.048 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.2.service' 2024-08-19T01:52:26.087 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:26.087 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-19T01:52:26.087 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-08-19T01:52:26.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.3 2024-08-19T01:52:26.410 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:26 smithi064 systemd[1]: Stopping Ceph osd.3 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:26.734 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:26 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:52:26.406+0000 7fec17bc1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:26.734 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:26 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:52:26.406+0000 7fec17bc1640 -1 osd.3 122 *** Got signal Terminated *** 2024-08-19T01:52:26.734 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:26 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:52:26.406+0000 7fec17bc1640 -1 osd.3 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:27.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:27 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:27.552+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:28.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:27.870+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:28.984 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:28 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:52:28.486+0000 7fec141d7640 -1 osd.3 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:06.052669+0000 front 2024-08-19T01:52:06.052640+0000 (oldest deadline 2024-08-19T01:52:27.752806+0000) 2024-08-19T01:52:28.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:28 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:28.590+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:29.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:28 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:28.840+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:29.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:29.572+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:29.734 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:29 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:52:29.474+0000 7fec141d7640 -1 osd.3 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:06.052669+0000 front 2024-08-19T01:52:06.052640+0000 (oldest deadline 2024-08-19T01:52:27.752806+0000) 2024-08-19T01:52:30.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:29 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:29.857+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:30.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:30.594+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:30.984 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3[36436]: 2024-08-19T01:52:30.489+0000 7fec141d7640 -1 osd.3 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:06.052669+0000 front 2024-08-19T01:52:06.052640+0000 (oldest deadline 2024-08-19T01:52:27.752806+0000) 2024-08-19T01:52:30.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:30 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:30.603+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:31.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:30 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:30.857+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:31.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:31.640+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:31.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:31 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:31.566+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:32.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:31 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:31.864+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:32.266 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:31 smithi064 podman[87002]: 2024-08-19 01:52:31.99175256 +0000 UTC m=+5.699650399 container died ccd8972e452982f679319c1e660125f17dbb4cb5aefa05ba9be1f7722778fdee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-de09dd0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-19T01:52:32.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:32.018+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:32.581 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:32 smithi064 podman[87002]: 2024-08-19 01:52:32.365091651 +0000 UTC m=+6.072989489 container cleanup ccd8972e452982f679319c1e660125f17dbb4cb5aefa05ba9be1f7722778fdee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-19T01:52:32.581 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:32 smithi064 bash[87002]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3 2024-08-19T01:52:32.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:32 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:32.659+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:32.984 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:32 smithi064 podman[87020]: 2024-08-19 01:52:32.658442864 +0000 UTC m=+0.660530623 container remove ccd8972e452982f679319c1e660125f17dbb4cb5aefa05ba9be1f7722778fdee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2) 2024-08-19T01:52:32.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:32 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:32.580+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:33.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:32 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:32.838+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:33.254 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:32 smithi064 podman[87104]: 2024-08-19 01:52:32.900388761 +0000 UTC m=+0.017379206 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:52:33.254 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:33 smithi064 podman[87104]: 2024-08-19 01:52:33.009415231 +0000 UTC m=+0.126405659 container create 42191e92d302f61f22c7ea7914478b1bc849a0a58e6264056d56a5e12e49b4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-19T01:52:33.254 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:33 smithi064 podman[87104]: 2024-08-19 01:52:33.253523902 +0000 UTC m=+0.370514350 container init 42191e92d302f61f22c7ea7914478b1bc849a0a58e6264056d56a5e12e49b4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-08-19T01:52:33.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:33.003+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:33.568 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:33 smithi064 podman[87104]: 2024-08-19 01:52:33.25707158 +0000 UTC m=+0.374062012 container start 42191e92d302f61f22c7ea7914478b1bc849a0a58e6264056d56a5e12e49b4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-08-19T01:52:33.568 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:33 smithi064 podman[87104]: 2024-08-19 01:52:33.345296202 +0000 UTC m=+0.462286652 container attach 42191e92d302f61f22c7ea7914478b1bc849a0a58e6264056d56a5e12e49b4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-08-19T01:52:33.568 INFO:journalctl@ceph.osd.3.smithi064.stdout:Aug 19 01:52:33 smithi064 podman[87104]: 2024-08-19 01:52:33.450604136 +0000 UTC m=+0.567594565 container died 42191e92d302f61f22c7ea7914478b1bc849a0a58e6264056d56a5e12e49b4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-19T01:52:33.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:33.616+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:33.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:33 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:33.567+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:34.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.3.service' 2024-08-19T01:52:34.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:34.122 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-08-19T01:52:34.122 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-08-19T01:52:34.122 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.4 2024-08-19T01:52:34.148 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:33 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:33.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:34.415 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:34 smithi123 systemd[1]: Stopping Ceph osd.4 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:34.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:34.004+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:34.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:34.629+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:34.986 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:34 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:34.569+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:35.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:34 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:34.797+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:35.165 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:34 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:52:34.708+0000 7fb399a16640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:35.166 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:34 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:52:34.708+0000 7fb399a16640 -1 osd.4 122 *** Got signal Terminated *** 2024-08-19T01:52:35.166 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:34 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:52:34.708+0000 7fb399a16640 -1 osd.4 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:35.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:35 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:35.000+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:35.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:35 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:35.606+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:35.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:35 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:35.572+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:36.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:35 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:35.749+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:36.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:35 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:35.957+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:36.731 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:36 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:52:36.649+0000 7fb392024640 -1 osd.4 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.667680+0000 front 2024-08-19T01:52:13.667695+0000 (oldest deadline 2024-08-19T01:52:35.967901+0000) 2024-08-19T01:52:36.731 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:36 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:36.465+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:36.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:36 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:36.557+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:36.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:36.572+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:36.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:36 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:36.572+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:37.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:36 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:36.729+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:37.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:36 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:36.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:37.742 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:37 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:52:37.649+0000 7fb392024640 -1 osd.4 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.667680+0000 front 2024-08-19T01:52:13.667695+0000 (oldest deadline 2024-08-19T01:52:35.967901+0000) 2024-08-19T01:52:37.742 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:37 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:37.465+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:37.948 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:37 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:37.513+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:37.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:37 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:37.529+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:37.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:37 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:37.529+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:38.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:37 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:37.740+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:38.374 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:37 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:37.954+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:38.374 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:37 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:37.954+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:38.698 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:38 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:38.472+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:38.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:38 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:38.373+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:38.780 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:38 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:52:38.653+0000 7fb392024640 -1 osd.4 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.667680+0000 front 2024-08-19T01:52:13.667695+0000 (oldest deadline 2024-08-19T01:52:35.967901+0000) 2024-08-19T01:52:38.780 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:38 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:38.438+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:38.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:38 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:38.509+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:38.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:38 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:38.509+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:39.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:38 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:38.778+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:39.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:38 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:38.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:39.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:38 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:38.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:39.701 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:39 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:39.513+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:39.701 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:39 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:39.513+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:39.701 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:39 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:39.348+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:39.746 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:39 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:39.346+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:39.746 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:39 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:39.505+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:39.746 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:39 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:39.505+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:39.787 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:39 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4[32730]: 2024-08-19T01:52:39.609+0000 7fb392024640 -1 osd.4 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.667680+0000 front 2024-08-19T01:52:13.667695+0000 (oldest deadline 2024-08-19T01:52:35.967901+0000) 2024-08-19T01:52:39.788 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:39 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:39.431+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:40.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:39 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:39.785+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:40.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:39 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:39.901+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:40.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:39 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:39.901+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:40.502 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:40 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:40.410+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:40.502 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:40 smithi123 podman[59353]: 2024-08-19 01:52:40.182616948 +0000 UTC m=+5.861906936 container died 05f86b31fc8666cb405fd8ffb55aaf79f4c3ddf397842773260ec0255c01593c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-19T01:52:40.698 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:40 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:40.509+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:40.699 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:40 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:40.509+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:40.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:40 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:40.398+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:40.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:40 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:40.383+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:40.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:40 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:40.507+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:40.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:40 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:40.507+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:40.780 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:40 smithi123 podman[59353]: 2024-08-19 01:52:40.600521295 +0000 UTC m=+6.279811276 container cleanup 05f86b31fc8666cb405fd8ffb55aaf79f4c3ddf397842773260ec0255c01593c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-08-19T01:52:40.780 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:40 smithi123 bash[59353]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4 2024-08-19T01:52:41.165 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:40 smithi123 podman[59368]: 2024-08-19 01:52:40.860200673 +0000 UTC m=+0.674572832 container remove 05f86b31fc8666cb405fd8ffb55aaf79f4c3ddf397842773260ec0255c01593c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-08-19T01:52:41.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:40 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:40.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:41.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:40 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:40.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:41.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:40 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:40.936+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:41.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:40 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:40.936+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:41.555 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:41 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:41.366+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:41.555 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:41 smithi123 podman[59463]: 2024-08-19 01:52:41.189570694 +0000 UTC m=+0.017107909 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:52:41.555 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:41 smithi123 podman[59463]: 2024-08-19 01:52:41.311096367 +0000 UTC m=+0.138633569 container create 5195c6a0ea3c550f59ceb86c4c23178735d5e2944658e1cf9f24a4d1930d25f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-19T01:52:41.556 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:41 smithi123 podman[59463]: 2024-08-19 01:52:41.555145645 +0000 UTC m=+0.382682851 container init 5195c6a0ea3c550f59ceb86c4c23178735d5e2944658e1cf9f24a4d1930d25f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:52:41.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:41 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:41.376+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:41.699 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:41 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:41.519+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:41.700 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:41 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:41.519+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:41.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:41 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:41.345+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:41.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:41 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:41.553+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:41.735 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:41 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:41.553+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:41.915 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:41 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:41.773+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:41.916 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:41 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:41.773+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:41.916 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:41 smithi123 podman[59463]: 2024-08-19 01:52:41.55855983 +0000 UTC m=+0.386097037 container start 5195c6a0ea3c550f59ceb86c4c23178735d5e2944658e1cf9f24a4d1930d25f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T01:52:41.916 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:41 smithi123 podman[59463]: 2024-08-19 01:52:41.655283774 +0000 UTC m=+0.482820979 container attach 5195c6a0ea3c550f59ceb86c4c23178735d5e2944658e1cf9f24a4d1930d25f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, GIT_CLEAN=True) 2024-08-19T01:52:41.916 INFO:journalctl@ceph.osd.4.smithi123.stdout:Aug 19 01:52:41 smithi123 podman[59463]: 2024-08-19 01:52:41.748712726 +0000 UTC m=+0.576249931 container died 5195c6a0ea3c550f59ceb86c4c23178735d5e2944658e1cf9f24a4d1930d25f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-08-19T01:52:42.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:41 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:41.928+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:42.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:41 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:41.928+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:42.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.4.service' 2024-08-19T01:52:42.524 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:42.525 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-08-19T01:52:42.525 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-19T01:52:42.525 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.5 2024-08-19T01:52:42.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:42 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:42.386+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:42.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:42.352+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:42.699 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:42.550+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:42.699 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:42.550+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:42.699 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:42 smithi161 systemd[1]: Stopping Ceph osd.5 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:42.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:42 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:42.332+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:42.735 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:42 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:42.532+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:42.735 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:42 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:42.532+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:43.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:42 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:42.807+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:43.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:42 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:42.807+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:43.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:42.973+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:43.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:42.973+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:43.199 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:42.864+0000 7f79f6f20640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:43.199 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:42.864+0000 7f79f6f20640 -1 osd.5 122 *** Got signal Terminated *** 2024-08-19T01:52:43.199 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:42 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:42.864+0000 7f79f6f20640 -1 osd.5 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:43.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:43.379+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:43.698 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:43.581+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:43.699 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:43.581+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:43.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:43.375+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:43.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:43 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:43.362+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:43.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:43 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:43.552+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:43.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:43 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:43.552+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:44.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:43.855+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:44.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:43 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:43.855+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:44.335 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:43.962+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:44.335 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:43 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:43.962+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:44.597 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:44 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:44.334+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:44.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:44 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:44.387+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:44.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:44 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:44.350+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:44.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:44 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:44.566+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:44.734 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:44 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:44.566+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:44.949 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:44 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:44.596+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:44.949 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:44 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:44.596+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:44.949 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:44 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:44.596+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:18.642932+0000 front 2024-08-19T01:52:18.642913+0000 (oldest deadline 2024-08-19T01:52:43.943125+0000) 2024-08-19T01:52:45.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:44 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:44.816+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:45.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:44 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:44.816+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:45.300 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:44 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:44.952+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:45.300 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:44 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:44.952+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:45.564 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:45 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:45.299+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:45.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:45 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:45.349+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:45.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:45 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:45.378+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:45.735 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:45 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:45.609+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:45.735 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:45 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:45.609+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:45.735 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:45 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:45.609+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:45.917 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:45 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:45.563+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:45.918 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:45 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:45.563+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:45.918 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:45 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:45.563+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:18.642932+0000 front 2024-08-19T01:52:18.642913+0000 (oldest deadline 2024-08-19T01:52:43.943125+0000) 2024-08-19T01:52:46.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:45 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:45.771+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:46.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:45 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:45.771+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:46.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:45 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:45.916+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:46.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:45 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:45.916+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:46.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:45 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:45.916+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:46.591 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:46.276+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:46.592 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:46.276+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:46.644 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:46 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:46.343+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:46.644 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:46 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:46.343+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:46.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:46 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:46.338+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:46.945 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:46.590+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:46.945 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:46.590+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:46.945 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:46.590+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:18.642932+0000 front 2024-08-19T01:52:18.642913+0000 (oldest deadline 2024-08-19T01:52:43.943125+0000) 2024-08-19T01:52:46.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:46 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:46.643+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:46.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:46 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:46.643+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:46.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:46 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:46.643+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:47.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:46 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:46.807+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:47.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:46 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:46.807+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:47.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:46.944+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:47.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:46.944+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:47.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:46 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:46.944+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:47.611 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:47.280+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:47.611 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:47.280+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:47.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:47 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:47.299+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:47.678 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:47 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:47.301+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:47.679 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:47 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:47.301+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:47.949 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:47.610+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.741177+0000 front 2024-08-19T01:52:05.741256+0000 (oldest deadline 2024-08-19T01:52:29.841314+0000) 2024-08-19T01:52:47.949 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:47.610+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.342414+0000 front 2024-08-19T01:52:13.342437+0000 (oldest deadline 2024-08-19T01:52:38.642570+0000) 2024-08-19T01:52:47.949 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5[33258]: 2024-08-19T01:52:47.610+0000 7f79f3536640 -1 osd.5 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:18.642932+0000 front 2024-08-19T01:52:18.642913+0000 (oldest deadline 2024-08-19T01:52:43.943125+0000) 2024-08-19T01:52:47.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:47 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:47.678+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:47.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:47 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:47.678+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:47.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:47 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:47.678+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:48.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:47 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:47.821+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:48.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:47 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:47.821+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:48.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:47 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:47.821+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:48.296 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:47.954+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:48.297 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:47.954+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:48.297 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:47 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:47.954+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:48.562 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:48 smithi161 podman[60440]: 2024-08-19 01:52:48.330192633 +0000 UTC m=+5.604768571 container died 874064f2c12796e93ab70016102019b47af1d509096e13b72e5166dc2bd08b9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T01:52:48.562 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:48 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:48.295+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:48.562 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:48 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:48.295+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:48.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:48 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:48.272+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:48.682 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:48 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:48.310+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:48.683 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:48 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:48.310+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:48.913 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:48 smithi161 podman[60440]: 2024-08-19 01:52:48.73189336 +0000 UTC m=+6.006469298 container cleanup 874064f2c12796e93ab70016102019b47af1d509096e13b72e5166dc2bd08b9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T01:52:48.913 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:48 smithi161 bash[60440]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5 2024-08-19T01:52:48.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:48 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:48.682+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:48.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:48 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:48.682+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:48.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:48 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:48.682+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:49.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:48 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:48.824+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:49.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:48 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:48.824+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:49.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:48 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:48.824+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:49.198 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:49 smithi161 podman[60461]: 2024-08-19 01:52:49.02556163 +0000 UTC m=+0.686375605 container remove 874064f2c12796e93ab70016102019b47af1d509096e13b72e5166dc2bd08b9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-19T01:52:49.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:48 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:48.912+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:49.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:48 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:48.912+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:49.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:48 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:48.912+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:49.592 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:49 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:49.339+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:49.592 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:49 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:49.339+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:49.592 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:49 smithi161 podman[60557]: 2024-08-19 01:52:49.327124425 +0000 UTC m=+0.023747700 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:52:49.592 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:49 smithi161 podman[60557]: 2024-08-19 01:52:49.460048828 +0000 UTC m=+0.156672102 container create 72674cf14bbb394e9570ad5a7299edf88c325bd8b9f1dde231a64aba04bedd81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-08-19T01:52:49.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:49 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:49.318+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:49.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:49 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:49.318+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:49.729 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:49 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:49.314+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:49.729 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:49 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:49.314+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:49.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:49 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:49.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:49.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:49 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:49.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:49.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:49 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:49.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:50.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:49 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:49.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:50.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:49 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:49.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:50.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:49 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:49.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:50.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:49 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:49.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:50.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:49 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:49.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:50.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:49 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:49.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:50.199 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:49 smithi161 podman[60557]: 2024-08-19 01:52:49.729495949 +0000 UTC m=+0.426119236 container init 72674cf14bbb394e9570ad5a7299edf88c325bd8b9f1dde231a64aba04bedd81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5-deactivate, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T01:52:50.199 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:49 smithi161 podman[60557]: 2024-08-19 01:52:49.732794877 +0000 UTC m=+0.429418151 container start 72674cf14bbb394e9570ad5a7299edf88c325bd8b9f1dde231a64aba04bedd81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:52:50.199 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:49 smithi161 podman[60557]: 2024-08-19 01:52:49.821083485 +0000 UTC m=+0.517706770 container attach 72674cf14bbb394e9570ad5a7299edf88c325bd8b9f1dde231a64aba04bedd81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-19T01:52:50.200 INFO:journalctl@ceph.osd.5.smithi161.stdout:Aug 19 01:52:49 smithi161 podman[60557]: 2024-08-19 01:52:49.91719326 +0000 UTC m=+0.613816542 container died 72674cf14bbb394e9570ad5a7299edf88c325bd8b9f1dde231a64aba04bedd81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-5-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T01:52:50.525 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:50 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:50.300+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:50.525 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:50 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:50.300+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:50.559 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.5.service' 2024-08-19T01:52:50.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:50.600 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-19T01:52:50.600 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-08-19T01:52:50.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.6 2024-08-19T01:52:50.627 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:50.285+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:50.627 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:50.285+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:50.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:50 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:50.289+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:50.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:50 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:50.289+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:50.927 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:50 smithi064 systemd[1]: Stopping Ceph osd.6 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:50.928 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:50.704+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:50.928 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:50.704+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:50.928 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:50.704+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:51.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:50 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:50.810+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:51.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:50 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:50.810+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:51.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:50 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:50.810+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:51.234 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:50.926+0000 7fc59dcea640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:51.234 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:50.926+0000 7fc59dcea640 -1 osd.6 122 *** Got signal Terminated *** 2024-08-19T01:52:51.234 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:50 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:50.926+0000 7fc59dcea640 -1 osd.6 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:51.269 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:50 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:50.964+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:51.269 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:50 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:50.964+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:51.269 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:50 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:50.964+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:51.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:51 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:51.268+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:51.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:51 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:51.268+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:51.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:51 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:51.267+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:51.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:51 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:51.267+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:51.717 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:51 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:51.297+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:51.717 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:51 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:51.297+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:51.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:51 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:51.716+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:51.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:51 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:51.716+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:51.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:51 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:51.716+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:52.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:51 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:51.822+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:52.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:51 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:51.822+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:52.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:51 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:51.822+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:52.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:51 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:51.929+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:52.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:51 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:51.929+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:52.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:51 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:51.929+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:52.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:52 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:52.237+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:52.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:52 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:52.237+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:52.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:52 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:52.299+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:52.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:52 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:52.299+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:52.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:52 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:52.299+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:52.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:52 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:52.254+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:52.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:52 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:52.254+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:53.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:52 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:52.790+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:53.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:52 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:52.790+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:53.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:52 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:52.790+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:53.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:52 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:52.790+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:52:53.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:52 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:52.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:53.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:52 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:52.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:53.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:52 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:52.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:53.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:52 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:52.764+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:53.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:52 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:52.764+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:53.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:52 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:52.764+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:53.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:53 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:53.215+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:53.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:53 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:53.215+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:53.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:53 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:53.215+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:52:53.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:53 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:53.337+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:53.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:53 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:53.337+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:53.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:53 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:53.337+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:53.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:53.287+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:53.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:53.287+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:54.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:53 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:53.782+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:54.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:53 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:53.782+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:54.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:53 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:53.782+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:54.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:53 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:53.782+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:52:54.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:53 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:53.871+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:54.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:53 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:53.871+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:54.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:53 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:53.871+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:54.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:53.767+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:54.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:53.767+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:54.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:53.767+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:54.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:53.767+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:52:54.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:54.249+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:54.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:54.249+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:54.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:54.249+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:52:54.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:54 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:54.312+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:54.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:54 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:54.312+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:54.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:54 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:54.312+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:54.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:54 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:54.243+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:54.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:54 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:54.243+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:55.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:54.806+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:55.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:54.806+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:55.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:54.806+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:55.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:54 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:54.806+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:52:55.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:54 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:54.868+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:55.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:54 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:54.868+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:55.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:54 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:54.868+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:55.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:54 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:54.790+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:55.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:54 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:54.790+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:55.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:54 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:54.790+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:55.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:54 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:54.790+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:52:55.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:55 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:55.268+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:55.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:55 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:55.268+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:55.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:55 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:55.268+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:52:55.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:55.263+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:55.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:55.263+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:55.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:55.263+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:55.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:55.237+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:11.419890+0000 front 2024-08-19T01:52:11.419873+0000 (oldest deadline 2024-08-19T01:52:38.420540+0000) 2024-08-19T01:52:55.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:55.237+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.121963+0000 front 2024-08-19T01:52:20.121773+0000 (oldest deadline 2024-08-19T01:52:45.421893+0000) 2024-08-19T01:52:55.734 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6[43296]: 2024-08-19T01:52:55.237+0000 7fc59a300640 -1 osd.6 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:28.823576+0000 front 2024-08-19T01:52:28.823607+0000 (oldest deadline 2024-08-19T01:52:54.723717+0000) 2024-08-19T01:52:56.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:55 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:55.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:56.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:55 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:55.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:56.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:55 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:55.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:56.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:55 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:55.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:52:56.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:55.842+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:56.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:55.842+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:56.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:55 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:55.842+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:56.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:55.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:56.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:55.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:56.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:55.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:56.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:55 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:55.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:52:56.661 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:56 smithi064 podman[87206]: 2024-08-19 01:52:56.370756141 +0000 UTC m=+5.551003517 container died 9adc479ddb88ce461a4a259d11a08ae72f3cdd56f709b4d97d30160e7f01d2db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:52:56.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:56.258+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:56.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:56.258+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:56.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:56.258+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:52:56.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:56 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:56.239+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:56.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:56 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:56.239+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:56.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:56 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:56.239+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:56.984 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:56 smithi064 podman[87206]: 2024-08-19 01:52:56.844215943 +0000 UTC m=+6.024463319 container cleanup 9adc479ddb88ce461a4a259d11a08ae72f3cdd56f709b4d97d30160e7f01d2db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-08-19T01:52:56.984 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:56 smithi064 bash[87206]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6 2024-08-19T01:52:56.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:56.755+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:56.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:56.755+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:56.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:56.755+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:56.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:56 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:56.755+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:52:57.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:56.797+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:57.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:56.797+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:57.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:56.797+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:57.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:56.797+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:52:57.192 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:56 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:56.827+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:57.193 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:56 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:56.827+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:57.193 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:56 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:56.827+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:57.428 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:57 smithi064 podman[87236]: 2024-08-19 01:52:57.087249054 +0000 UTC m=+0.714818708 container remove 9adc479ddb88ce461a4a259d11a08ae72f3cdd56f709b4d97d30160e7f01d2db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T01:52:57.448 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:57 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:57.191+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:57.448 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:57 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:57.191+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:57.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:57 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:57.191+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:57.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:57.227+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:57.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:57.227+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:57.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:57.227+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:52:57.707 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:57 smithi064 podman[87324]: 2024-08-19 01:52:57.330231798 +0000 UTC m=+0.017282907 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:52:57.708 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:57 smithi064 podman[87324]: 2024-08-19 01:52:57.446563654 +0000 UTC m=+0.133614785 container create a1f14158bee3e6ca416b6e289d686587ea3a1bf021116b184c52a4ce718e55e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731) 2024-08-19T01:52:57.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:57.776+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:57.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:57.776+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:57.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:57.776+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:57.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:57 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:57.776+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:52:57.985 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:57 smithi064 podman[87324]: 2024-08-19 01:52:57.707486911 +0000 UTC m=+0.394538012 container init a1f14158bee3e6ca416b6e289d686587ea3a1bf021116b184c52a4ce718e55e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-08-19T01:52:57.985 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:57 smithi064 podman[87324]: 2024-08-19 01:52:57.711280058 +0000 UTC m=+0.398331151 container start a1f14158bee3e6ca416b6e289d686587ea3a1bf021116b184c52a4ce718e55e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T01:52:57.985 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:57 smithi064 podman[87324]: 2024-08-19 01:52:57.815965907 +0000 UTC m=+0.503017011 container attach a1f14158bee3e6ca416b6e289d686587ea3a1bf021116b184c52a4ce718e55e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-19T01:52:57.985 INFO:journalctl@ceph.osd.6.smithi064.stdout:Aug 19 01:52:57 smithi064 podman[87324]: 2024-08-19 01:52:57.901329377 +0000 UTC m=+0.588380470 container died a1f14158bee3e6ca416b6e289d686587ea3a1bf021116b184c52a4ce718e55e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-08-19T01:52:58.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:57.836+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:58.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:57.836+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:58.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:57.836+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:58.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:57 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:57.836+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:52:58.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:57 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:57.877+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:58.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:57 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:57.877+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:58.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:57 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:57.877+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:58.433 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:58.270+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:58.433 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:58.270+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:58.433 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:58.270+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:52:58.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.6.service' 2024-08-19T01:52:58.600 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:52:58.601 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-08-19T01:52:58.601 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-08-19T01:52:58.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.7 2024-08-19T01:52:58.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:58 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:58.224+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:58.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:58 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:58.224+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:58.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:58 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:58.224+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:58.825 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:58 smithi123 systemd[1]: Stopping Ceph osd.7 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:52:59.165 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:58.919+0000 7f6ad6c3d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:52:59.165 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:58.919+0000 7f6ad6c3d640 -1 osd.7 122 *** Got signal Terminated *** 2024-08-19T01:52:59.166 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:58.919+0000 7f6ad6c3d640 -1 osd.7 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:52:59.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:58.824+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:52:59.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:58.824+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:52:59.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:58.824+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:52:59.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:58 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:58.824+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:52:59.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:58 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:58.843+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:59.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:58 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:58.843+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:59.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:58 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:58.843+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:52:59.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:58 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:58.806+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:52:59.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:58 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:58.806+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:52:59.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:58 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:58.806+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:52:59.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:58 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:58.806+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:52:59.594 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:59.241+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:52:59.594 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:59.241+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:52:59.595 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:52:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:52:59.241+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:52:59.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:59 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:59.269+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:52:59.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:59 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:59.269+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:52:59.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:52:59 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:52:59.269+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:52:59.948 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:59.836+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:52:59.949 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:59.836+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:52:59.949 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:52:59 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:52:59.836+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:00.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:59 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:59.810+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:00.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:59 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:59.810+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:00.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:59 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:59.810+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:00.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:52:59 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:52:59.810+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:00.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:59 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:59.781+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:00.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:59 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:59.781+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:00.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:59 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:59.781+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:00.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:52:59 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:52:59.781+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:00.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:00.279+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:53:00.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:00.279+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:53:00.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:00.279+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:53:00.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:00 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:00.218+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:00.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:00 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:00.218+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:00.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:00 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:00.218+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:01.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:00.764+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:01.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:00.764+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:01.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:00.764+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:01.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:00.764+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:01.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:00 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:00.883+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:01.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:00 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:00.883+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:01.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:00 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:00.883+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:01.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:00.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:01.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:00.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:01.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:00.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:01.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:00.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:01.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:00 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:00.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:01.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:01 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:01.311+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:53:01.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:01 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:01.311+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:53:01.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:01 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:01.311+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:53:01.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:01 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:01.198+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:01.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:01 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:01.198+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:01.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:01 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:01.198+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:02.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:01 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:01.747+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:02.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:01 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:01.747+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:02.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:01 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:01.747+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:02.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:01 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:01.747+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:02.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:01 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:01.908+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:02.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:01 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:01.908+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:02.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:01 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:01.908+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:02.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:01.864+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:02.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:01.864+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:02.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:01.864+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:02.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:01.864+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:02.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:01 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:01.864+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:02.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:02.289+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:53:02.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:02.289+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:53:02.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:02.289+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:53:02.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:02.289+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:37.506688+0000 front 2024-08-19T01:52:37.506710+0000 (oldest deadline 2024-08-19T01:53:01.606878+0000) 2024-08-19T01:53:02.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:02 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:02.226+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:02.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:02 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:02.226+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:02.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:02 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:02.226+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:02.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:02 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:02.226+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:03.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:02.793+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:03.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:02.793+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:03.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:02.793+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:03.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:02 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:02.793+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:03.189 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:02 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:02.921+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:03.190 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:02 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:02.921+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:03.190 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:02 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:02.921+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:03.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:02 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:02.877+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:03.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:02 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:02.877+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:03.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:02 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:02.877+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:03.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:02 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:02.877+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:03.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:02 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:02.877+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:03.448 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:03 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:03.188+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:03.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:03 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:03.188+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:03.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:03 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:03.188+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:03.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:03 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:03.188+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:03.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:03.281+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.800084+0000 front 2024-08-19T01:52:12.800202+0000 (oldest deadline 2024-08-19T01:52:35.700307+0000) 2024-08-19T01:53:03.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:03.281+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:21.402587+0000 front 2024-08-19T01:52:21.402405+0000 (oldest deadline 2024-08-19T01:52:48.403120+0000) 2024-08-19T01:53:03.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:03.281+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:30.604947+0000 front 2024-08-19T01:52:30.604836+0000 (oldest deadline 2024-08-19T01:52:52.304925+0000) 2024-08-19T01:53:03.666 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7[39654]: 2024-08-19T01:53:03.281+0000 7f6acf24b640 -1 osd.7 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:37.506688+0000 front 2024-08-19T01:52:37.506710+0000 (oldest deadline 2024-08-19T01:53:01.606878+0000) 2024-08-19T01:53:04.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:03.774+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:04.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:03.774+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:04.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:03.774+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:04.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:03 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:03.774+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:04.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:03 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:03.918+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:04.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:03 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:03.918+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:04.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:03 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:03.918+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:04.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:03 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:03.875+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:04.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:03 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:03.875+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:04.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:03 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:03.875+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:04.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:03 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:03.875+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:04.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:03 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:03.875+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:04.665 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:04 smithi123 podman[59573]: 2024-08-19 01:53:04.379367192 +0000 UTC m=+5.563664703 container died e701a333bd3ee4049861b677b8e8c98bf7635ea43ee8fd00a32e402ca78c6312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-08-19T01:53:04.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:04.213+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:04.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:04.213+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:04.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:04.213+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:04.699 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:04.213+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:05.021 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:04 smithi123 podman[59573]: 2024-08-19 01:53:04.778254426 +0000 UTC m=+5.962551924 container cleanup e701a333bd3ee4049861b677b8e8c98bf7635ea43ee8fd00a32e402ca78c6312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-19T01:53:05.021 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:04 smithi123 bash[59573]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7 2024-08-19T01:53:05.022 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:04 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:04.760+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:05.022 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:04 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:04.760+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:05.022 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:04 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:04.760+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:05.022 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:04 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:04.760+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:05.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:05.165+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:05.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:05.165+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:05.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:05.165+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:05.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:05.165+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:05.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:04.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:05.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:04.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:05.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:04.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:05.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:04 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:04.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:05.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:04 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:04.836+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:05.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:04 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:04.836+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:05.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:04 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:04.836+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:05.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:04 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:04.836+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:05.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:04 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:04.836+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:05.353 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:05 smithi123 podman[59591]: 2024-08-19 01:53:05.021196098 +0000 UTC m=+0.639887330 container remove e701a333bd3ee4049861b677b8e8c98bf7635ea43ee8fd00a32e402ca78c6312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-08-19T01:53:05.616 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:05 smithi123 podman[59684]: 2024-08-19 01:53:05.255865509 +0000 UTC m=+0.017002427 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:53:05.617 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:05 smithi123 podman[59684]: 2024-08-19 01:53:05.380504858 +0000 UTC m=+0.141641783 container create cf482df6eb2950d66f5e7d699ea018067758222c7776acbbe50d5def3bfd4008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-19T01:53:05.617 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:05 smithi123 podman[59684]: 2024-08-19 01:53:05.616213136 +0000 UTC m=+0.377350068 container init cf482df6eb2950d66f5e7d699ea018067758222c7776acbbe50d5def3bfd4008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:53:05.915 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:05 smithi123 podman[59684]: 2024-08-19 01:53:05.619807222 +0000 UTC m=+0.380944141 container start cf482df6eb2950d66f5e7d699ea018067758222c7776acbbe50d5def3bfd4008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:53:05.916 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:05 smithi123 podman[59684]: 2024-08-19 01:53:05.724595467 +0000 UTC m=+0.485732385 container attach cf482df6eb2950d66f5e7d699ea018067758222c7776acbbe50d5def3bfd4008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7-deactivate, RELEASE=squid-de09dd0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-08-19T01:53:05.916 INFO:journalctl@ceph.osd.7.smithi123.stdout:Aug 19 01:53:05 smithi123 podman[59684]: 2024-08-19 01:53:05.80751906 +0000 UTC m=+0.568655980 container died cf482df6eb2950d66f5e7d699ea018067758222c7776acbbe50d5def3bfd4008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-08-19T01:53:05.916 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:05 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:05.755+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:05.916 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:05 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:05.755+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:05.916 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:05 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:05.755+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:05.916 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:05 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:05.755+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:05.916 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:05 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:05.755+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:06.146 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:05.875+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:06.146 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:05.875+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:06.147 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:05.875+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:06.147 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:05 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:05.875+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:06.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:05.880+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:06.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:05.880+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:06.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:05.880+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:06.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:05.880+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:06.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:05 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:05.880+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:06.448 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:06.146+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:06.448 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:06.146+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:06.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:06.146+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:06.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:06.146+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:06.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.7.service' 2024-08-19T01:53:06.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:06.540 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-08-19T01:53:06.541 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-08-19T01:53:06.541 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.8 2024-08-19T01:53:06.853 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 systemd[1]: Stopping Ceph osd.8 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:07.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:06 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:06.724+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:07.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:06 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:06.724+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:07.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:06 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:06.724+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:07.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:06 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:06.724+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:07.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:06 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:06.724+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:07.166 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:06.851+0000 7fde8d922640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:07.166 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:06.851+0000 7fde8d922640 -1 osd.8 122 *** Got signal Terminated *** 2024-08-19T01:53:07.166 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:06.851+0000 7fde8d922640 -1 osd.8 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:53:07.167 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:06.889+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:07.167 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:06.889+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:07.167 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:06.889+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:07.167 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:06 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:06.889+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:07.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:06.862+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:07.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:06.862+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:07.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:06.862+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:07.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:06.862+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:07.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:06 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:06.862+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:07.448 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:07.164+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:07.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:07.164+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:07.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:07.164+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:07.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:07.164+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:08.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:07 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:07.768+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:08.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:07 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:07.768+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:08.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:07 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:07.768+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:08.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:07 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:07.768+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:08.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:07 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:07.768+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:08.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:07.936+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:08.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:07.936+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:08.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:07.936+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:08.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:07 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:07.936+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:08.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:08.119+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:08.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:08.119+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:08.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:08.119+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:08.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:08.119+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:08.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:07.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:08.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:07.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:08.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:07.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:08.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:07.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:08.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:07 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:07.817+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:08.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:08.780+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:08.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:08.780+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:08.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:08.780+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:08.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:08.780+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:08.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:08 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:08.780+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:09.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:08.722+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:09.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:08.722+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:09.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:08.722+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:09.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:08.722+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:09.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:08.722+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:09.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:08.951+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:09.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:08.951+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:09.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:08.951+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:09.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:08 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:08.951+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:09.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:09.126+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:09.450 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:09.126+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:09.450 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:09.126+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:09.450 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:09.126+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:09.450 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:09.126+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:42.632513+0000 front 2024-08-19T01:52:42.632482+0000 (oldest deadline 2024-08-19T01:53:09.033187+0000) 2024-08-19T01:53:10.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:09.766+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:10.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:09.766+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:10.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:09.766+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:10.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:09.766+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:10.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:09 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:09.766+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:10.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:09.804+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:10.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:09.804+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:10.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:09.804+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:10.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:09.804+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:10.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:09 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:09.804+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:10.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:09.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:10.755 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:09.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:10.755 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:09.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:10.755 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:09.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:10.755 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:09 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:09.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:10.755 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:10.133+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:10.756 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:10.133+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:10.756 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:10.133+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:10.756 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:10.133+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:10.756 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:10.133+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:42.632513+0000 front 2024-08-19T01:52:42.632482+0000 (oldest deadline 2024-08-19T01:53:09.033187+0000) 2024-08-19T01:53:11.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:10.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:11.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:10.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:11.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:10.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:11.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:10.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:11.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:10 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:10.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:11.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:10.805+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:11.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:10.805+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:11.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:10.805+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:11.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:10.805+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:11.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:10 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:10.805+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:11.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:11.167+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.827428+0000 front 2024-08-19T01:52:14.827567+0000 (oldest deadline 2024-08-19T01:52:37.727513+0000) 2024-08-19T01:53:11.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:11.167+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.930061+0000 front 2024-08-19T01:52:20.930095+0000 (oldest deadline 2024-08-19T01:52:46.230181+0000) 2024-08-19T01:53:11.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:11.167+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:26.230789+0000 front 2024-08-19T01:52:26.230743+0000 (oldest deadline 2024-08-19T01:52:52.130616+0000) 2024-08-19T01:53:11.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:11.167+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.031637+0000 front 2024-08-19T01:52:38.031621+0000 (oldest deadline 2024-08-19T01:53:01.531585+0000) 2024-08-19T01:53:11.449 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:11 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8[40139]: 2024-08-19T01:53:11.167+0000 7fde89f38640 -1 osd.8 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:42.632513+0000 front 2024-08-19T01:52:42.632482+0000 (oldest deadline 2024-08-19T01:53:09.033187+0000) 2024-08-19T01:53:11.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:10.993+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:11.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:10.993+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:11.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:10.993+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:11.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:10.993+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:11.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:10 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:10.993+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:12.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:11.783+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:12.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:11.783+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:12.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:11.783+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:12.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:11.783+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:12.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:11 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:11.783+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:12.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:11.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:12.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:11.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:12.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:11.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:12.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:11.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:12.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:11.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:12.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:11 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:11.795+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:12.398 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:12.041+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:12.398 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:12.041+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:12.398 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:12.041+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:12.398 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:12.041+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:12.398 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:12 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:12.041+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:12.698 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:12 smithi161 podman[60663]: 2024-08-19 01:53:12.397595238 +0000 UTC m=+5.646333189 container died 3fcdaa503a821bb893e14bafaaf9dbdc5e0220c43e3b6d4348b1a296fd4919be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-19T01:53:13.002 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:12 smithi161 podman[60663]: 2024-08-19 01:53:12.810866836 +0000 UTC m=+6.059604778 container cleanup 3fcdaa503a821bb893e14bafaaf9dbdc5e0220c43e3b6d4348b1a296fd4919be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-19T01:53:13.003 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:12 smithi161 bash[60663]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8 2024-08-19T01:53:13.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:12.798+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:13.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:12.798+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:13.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:12.798+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:13.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:12.798+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:13.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:12 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:12.798+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:13.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:12.807+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:13.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:12.807+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:13.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:12.808+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:13.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:12.808+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:13.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:12.808+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:13.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:12 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:12.808+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:13.448 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:13 smithi161 podman[60678]: 2024-08-19 01:53:13.015323437 +0000 UTC m=+0.618615140 container remove 3fcdaa503a821bb893e14bafaaf9dbdc5e0220c43e3b6d4348b1a296fd4919be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:53:13.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.001+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:13.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.001+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:13.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.001+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:13.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.001+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:13.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.001+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:13.724 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:13 smithi161 podman[60768]: 2024-08-19 01:53:13.35432502 +0000 UTC m=+0.016952636 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:53:13.724 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:13 smithi161 podman[60768]: 2024-08-19 01:53:13.488469126 +0000 UTC m=+0.151096770 container create 443bf7d90a8aae8ff81baaff06c2fb3ccd8d02bdb84da0f4a1aa06e800b70815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-19T01:53:13.725 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:13 smithi161 podman[60768]: 2024-08-19 01:53:13.724352483 +0000 UTC m=+0.386980098 container init 443bf7d90a8aae8ff81baaff06c2fb3ccd8d02bdb84da0f4a1aa06e800b70815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:53:14.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:13.776+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:14.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:13.776+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:14.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:13.776+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:14.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:13.776+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:14.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:13.776+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:14.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:13 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:13.776+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:14.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:13 smithi161 podman[60768]: 2024-08-19 01:53:13.727905701 +0000 UTC m=+0.390533314 container start 443bf7d90a8aae8ff81baaff06c2fb3ccd8d02bdb84da0f4a1aa06e800b70815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-19T01:53:14.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:13 smithi161 podman[60768]: 2024-08-19 01:53:13.832594483 +0000 UTC m=+0.495222111 container attach 443bf7d90a8aae8ff81baaff06c2fb3ccd8d02bdb84da0f4a1aa06e800b70815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731) 2024-08-19T01:53:14.199 INFO:journalctl@ceph.osd.8.smithi161.stdout:Aug 19 01:53:13 smithi161 podman[60768]: 2024-08-19 01:53:13.920270152 +0000 UTC m=+0.582897765 container died 443bf7d90a8aae8ff81baaff06c2fb3ccd8d02bdb84da0f4a1aa06e800b70815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:53:14.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.972+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:14.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.972+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:14.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.972+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:14.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.972+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:14.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:13.972+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:14.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:13.801+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:14.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:13.801+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:14.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:13.801+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:14.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:13.801+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:14.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:13.801+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:14.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:13 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:13.801+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:14.619 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.8.service' 2024-08-19T01:53:14.660 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:14.660 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-08-19T01:53:14.660 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-08-19T01:53:14.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.9 2024-08-19T01:53:14.981 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 systemd[1]: Stopping Ceph osd.9 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:14.981 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.785+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:14.981 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.785+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:14.981 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.785+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:14.981 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.785+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:14.982 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.785+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:14.982 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.785+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:14.982 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.980+0000 7fae7d73f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:14.982 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.980+0000 7fae7d73f640 -1 osd.9 122 *** Got signal Terminated *** 2024-08-19T01:53:14.982 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:14 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:14.980+0000 7fae7d73f640 -1 osd.9 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:53:15.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:14 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:14.775+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:15.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:14 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:14.775+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:15.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:14 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:14.775+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:15.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:14 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:14.775+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:15.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:14 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:14.775+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:15.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:14 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:14.775+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:15.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:14 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:14.976+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:15.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:14 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:14.976+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:15.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:14 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:14.976+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:15.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:14 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:14.976+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:15.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:14 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:14.976+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:16.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:15 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:15.818+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:16.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:15 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:15.818+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:16.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:15 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:15.818+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:16.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:15 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:15.818+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:16.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:15 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:15.818+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:16.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:15 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:15.818+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:16.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:15 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:15.933+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:16.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:15 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:15.933+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:16.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:15 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:15.933+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:16.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:15 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:15.933+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:16.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:15 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:15.933+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:16.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:15.747+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:16.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:15.747+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:16.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:15.747+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:16.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:15.747+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:16.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:15.747+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:16.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:15 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:15.747+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:16.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:16 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:16.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:16.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:16 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:16.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:16.984 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:16 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:16.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:16.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:16 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:16.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:16.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:16 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:16.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:16.985 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:16 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:16.728+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:17.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:16 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:16.854+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:17.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:16 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:16.854+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:17.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:16 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:16.854+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:17.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:16 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:16.854+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:17.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:16 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:16.854+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:17.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:16 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:16.854+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:17.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:16 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:16.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:17.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:16 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:16.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:17.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:16 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:16.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:17.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:16 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:16.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:17.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:16 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:16.923+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:18.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:17 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:17.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:18.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:17 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:17.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:18.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:17 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:17.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:18.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:17 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:17.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:18.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:17 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:17.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:18.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:17 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:17.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:18.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:17 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:17.808+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:18.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:17 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:17.925+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:18.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:17 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:17.925+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:18.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:17 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:17.925+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:18.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:17 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:17.925+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:18.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:17 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:17.925+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:18.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:17 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:17.756+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:18.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:17 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:17.756+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:18.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:17 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:17.756+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:18.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:17 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:17.756+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:18.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:17 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:17.756+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:18.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:17 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:17.756+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:19.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:18 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:18.803+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:19.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:18 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:18.803+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:19.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:18 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:18.803+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:19.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:18 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:18.803+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:19.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:18 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:18.803+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:19.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:18 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:18.803+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:19.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:18 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:18.803+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:19.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:18 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:18.735+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:19.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:18 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:18.735+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:19.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:18 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:18.735+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:19.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:18 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:18.735+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:19.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:18 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:18.735+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:19.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:18 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:18.735+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:19.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:18.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:19.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:18.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:19.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:18.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:19.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:18.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:19.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:18 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:18.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:20.135 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:19 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:19.746+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:04.111156+0000 front 2024-08-19T01:52:04.111194+0000 (oldest deadline 2024-08-19T01:52:27.011398+0000) 2024-08-19T01:53:20.136 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:19 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:19.746+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:12.812904+0000 front 2024-08-19T01:52:12.813083+0000 (oldest deadline 2024-08-19T01:52:36.313120+0000) 2024-08-19T01:53:20.136 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:19 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:19.746+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.315423+0000 front 2024-08-19T01:52:20.315633+0000 (oldest deadline 2024-08-19T01:52:45.015296+0000) 2024-08-19T01:53:20.136 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:19 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:19.746+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:31.318014+0000 front 2024-08-19T01:52:31.317991+0000 (oldest deadline 2024-08-19T01:52:53.618211+0000) 2024-08-19T01:53:20.136 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:19 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:19.746+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:34.719339+0000 front 2024-08-19T01:52:34.719436+0000 (oldest deadline 2024-08-19T01:53:00.619400+0000) 2024-08-19T01:53:20.136 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:19 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9[49781]: 2024-08-19T01:53:19.746+0000 7fae75d4d640 -1 osd.9 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:46.321617+0000 front 2024-08-19T01:52:46.321736+0000 (oldest deadline 2024-08-19T01:53:11.621435+0000) 2024-08-19T01:53:20.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:19.801+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:20.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:19.801+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:20.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:19.801+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:20.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:19.801+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:20.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:19.801+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:20.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:19.801+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:20.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:19 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:19.801+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:20.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:19 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:19.940+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:20.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:19 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:19.940+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:20.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:19 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:19.940+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:20.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:19 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:19.940+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:20.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:19 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:19.940+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:20.848 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:20 smithi064 podman[87425]: 2024-08-19 01:53:20.592739587 +0000 UTC m=+5.725751245 container died 9bfbeaeec7d4cf97b42a4a4dbca76db114b08285d4201e9103b5b9b0023305d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-08-19T01:53:21.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:20 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:20.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:21.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:20 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:20.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:21.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:20 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:20.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:21.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:20 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:20.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:21.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:20 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:20.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:21.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:20 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:20.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:21.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:20 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:20.799+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:21.190 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:21 smithi064 podman[87425]: 2024-08-19 01:53:21.013392656 +0000 UTC m=+6.146404305 container cleanup 9bfbeaeec7d4cf97b42a4a4dbca76db114b08285d4201e9103b5b9b0023305d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T01:53:21.190 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:21 smithi064 bash[87425]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9 2024-08-19T01:53:21.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:20.909+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:21.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:20.909+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:21.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:20.909+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:21.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:20.909+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:21.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:20.909+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:21.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:20.909+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:21.484 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:21 smithi064 podman[87441]: 2024-08-19 01:53:21.189782959 +0000 UTC m=+0.598519333 container remove 9bfbeaeec7d4cf97b42a4a4dbca76db114b08285d4201e9103b5b9b0023305d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-19T01:53:21.943 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:21 smithi064 podman[87538]: 2024-08-19 01:53:21.583976089 +0000 UTC m=+0.017423990 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:53:21.944 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:21 smithi064 podman[87538]: 2024-08-19 01:53:21.69067604 +0000 UTC m=+0.124123929 container create 5d836631f47889ac167f272b95e86cf1767c9bec4d3d4eaffaee77214562df03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS) 2024-08-19T01:53:21.944 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:21 smithi064 podman[87538]: 2024-08-19 01:53:21.943328421 +0000 UTC m=+0.376776321 container init 5d836631f47889ac167f272b95e86cf1767c9bec4d3d4eaffaee77214562df03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9-deactivate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True) 2024-08-19T01:53:22.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:21.796+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:22.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:21.796+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:22.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:21.796+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:22.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:21.796+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:22.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:21.796+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:22.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:21.796+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:22.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:21 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:21.796+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:22.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:21 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:21.862+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:22.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:21 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:21.862+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:22.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:21 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:21.862+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:22.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:21 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:21.862+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:22.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:21 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:21.862+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:22.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:21 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:21.862+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:22.234 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:21 smithi064 podman[87538]: 2024-08-19 01:53:21.946875048 +0000 UTC m=+0.380322930 container start 5d836631f47889ac167f272b95e86cf1767c9bec4d3d4eaffaee77214562df03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=squid-de09dd0) 2024-08-19T01:53:22.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:22 smithi064 podman[87538]: 2024-08-19 01:53:22.018387046 +0000 UTC m=+0.451834929 container attach 5d836631f47889ac167f272b95e86cf1767c9bec4d3d4eaffaee77214562df03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9-deactivate, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-08-19T01:53:22.235 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:22 smithi064 podman[87538]: 2024-08-19 01:53:22.139249717 +0000 UTC m=+0.572697611 container died 5d836631f47889ac167f272b95e86cf1767c9bec4d3d4eaffaee77214562df03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-19T01:53:22.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.9.service' 2024-08-19T01:53:22.662 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:22 smithi064 podman[87562]: 2024-08-19 01:53:22.621986514 +0000 UTC m=+0.476989415 container remove 5d836631f47889ac167f272b95e86cf1767c9bec4d3d4eaffaee77214562df03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-08-19T01:53:22.662 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:22 smithi064 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.9.service: Deactivated successfully. 2024-08-19T01:53:22.662 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:22 smithi064 systemd[1]: Stopped Ceph osd.9 for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:53:22.662 INFO:journalctl@ceph.osd.9.smithi064.stdout:Aug 19 01:53:22 smithi064 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.9.service: Consumed 3min 46.580s CPU time. 2024-08-19T01:53:22.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:22.678 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-08-19T01:53:22.678 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-08-19T01:53:22.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.10 2024-08-19T01:53:22.991 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 systemd[1]: Stopping Ceph osd.10 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:22.991 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:22.991 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:22.992 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:22.992 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:22.992 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:22.992 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:22.992 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.809+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:22.992 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.990+0000 7fa90cb7a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:22.992 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.990+0000 7fa90cb7a640 -1 osd.10 122 *** Got signal Terminated *** 2024-08-19T01:53:22.993 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:22 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:22.990+0000 7fa90cb7a640 -1 osd.10 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:53:23.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:22 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:22.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:23.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:22 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:22.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:23.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:22 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:22.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:23.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:22 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:22.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:23.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:22 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:22.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:23.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:22 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:22.893+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:24.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:23 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:23.819+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:24.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:23 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:23.819+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:24.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:23 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:23.819+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:24.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:23 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:23.819+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:24.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:23 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:23.819+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:24.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:23 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:23.819+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:24.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:23 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:23.819+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:24.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:23 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:23.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:24.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:23 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:23.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:24.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:23 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:23.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:24.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:23 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:23.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:24.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:23 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:23.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:24.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:23 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:23.906+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:25.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:24 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:24.780+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:25.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:24 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:24.780+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:25.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:24 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:24.780+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:25.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:24 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:24.780+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:25.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:24 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:24.780+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:25.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:24 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:24.780+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:25.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:24 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:24.780+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:25.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:24.856+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:25.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:24.856+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:25.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:24.856+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:25.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:24.856+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:25.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:24.856+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:25.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:24.856+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:26.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:25 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:25.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:26.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:25 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:25.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:26.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:25 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:25.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:26.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:25 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:25.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:26.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:25 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:25.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:26.167 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:25 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:25.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:26.167 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:25 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:25.779+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:26.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:25 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:25.872+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:26.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:25 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:25.872+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:26.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:25 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:25.872+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:26.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:25 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:25.872+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:26.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:25 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:25.872+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:26.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:25 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:25.872+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:27.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:26 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:26.805+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:27.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:26 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:26.805+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:27.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:26 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:26.805+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:27.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:26 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:26.805+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:27.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:26 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:26.805+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:27.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:26 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:26.805+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:27.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:26 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:26.805+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:27.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:26 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:26.844+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:27.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:26 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:26.844+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:27.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:26 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:26.844+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:27.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:26 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:26.844+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:27.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:26 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:26.844+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:27.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:26 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:26.844+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:27.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:26 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:26.844+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:28.086 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:27.823+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:02.969849+0000 front 2024-08-19T01:52:02.970086+0000 (oldest deadline 2024-08-19T01:52:27.070021+0000) 2024-08-19T01:53:28.087 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:27.823+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:14.572328+0000 front 2024-08-19T01:52:14.572286+0000 (oldest deadline 2024-08-19T01:52:39.872476+0000) 2024-08-19T01:53:28.087 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:27.823+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:22.074241+0000 front 2024-08-19T01:52:22.074215+0000 (oldest deadline 2024-08-19T01:52:47.275058+0000) 2024-08-19T01:53:28.087 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:27.823+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-08-19T01:52:29.576140+0000 front 2024-08-19T01:52:29.576056+0000 (oldest deadline 2024-08-19T01:52:52.476044+0000) 2024-08-19T01:53:28.087 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:27.823+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:39.477615+0000 front 2024-08-19T01:52:39.477632+0000 (oldest deadline 2024-08-19T01:53:05.377792+0000) 2024-08-19T01:53:28.087 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:27.823+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:45.378382+0000 front 2024-08-19T01:52:45.378450+0000 (oldest deadline 2024-08-19T01:53:12.978879+0000) 2024-08-19T01:53:28.088 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:27 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10[46548]: 2024-08-19T01:53:27.823+0000 7fa905188640 -1 osd.10 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.079919+0000 front 2024-08-19T01:52:54.079991+0000 (oldest deadline 2024-08-19T01:53:16.979923+0000) 2024-08-19T01:53:28.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:27.805+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:28.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:27.805+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:28.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:27.805+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:28.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:27.805+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:28.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:27.805+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:28.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:27.805+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:28.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:27 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:27.805+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:28.665 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:28 smithi123 podman[59787]: 2024-08-19 01:53:28.539212822 +0000 UTC m=+5.657832530 container died 121271c48962df6855a81799705c467f3e7225c4aef0255041bdf2b2650f6008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-08-19T01:53:29.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:28 smithi123 podman[59787]: 2024-08-19 01:53:28.898219644 +0000 UTC m=+6.016839337 container cleanup 121271c48962df6855a81799705c467f3e7225c4aef0255041bdf2b2650f6008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, io.buildah.version=1.36.0) 2024-08-19T01:53:29.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:28 smithi123 bash[59787]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10 2024-08-19T01:53:29.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:28 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:28.838+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:29.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:28 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:28.838+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:29.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:28 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:28.838+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:29.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:28 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:28.838+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:29.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:28 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:28.838+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:29.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:28 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:28.838+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:29.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:28 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:28.838+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:29.526 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:29 smithi123 podman[59811]: 2024-08-19 01:53:29.182722949 +0000 UTC m=+0.640759298 container remove 121271c48962df6855a81799705c467f3e7225c4aef0255041bdf2b2650f6008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-19T01:53:29.836 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:29 smithi123 podman[59901]: 2024-08-19 01:53:29.428749273 +0000 UTC m=+0.017093883 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:53:29.837 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:29 smithi123 podman[59901]: 2024-08-19 01:53:29.533420098 +0000 UTC m=+0.121764701 container create 8cde6114f5fe69484ea00b95cbea169ae04f38d9ac9249b4db8b47a97daf4159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-08-19T01:53:29.837 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:29 smithi123 podman[59901]: 2024-08-19 01:53:29.752552424 +0000 UTC m=+0.340897028 container init 8cde6114f5fe69484ea00b95cbea169ae04f38d9ac9249b4db8b47a97daf4159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-08-19T01:53:29.837 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:29 smithi123 podman[59901]: 2024-08-19 01:53:29.755733107 +0000 UTC m=+0.344077720 container start 8cde6114f5fe69484ea00b95cbea169ae04f38d9ac9249b4db8b47a97daf4159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-08-19T01:53:30.165 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:29 smithi123 podman[59901]: 2024-08-19 01:53:29.835962447 +0000 UTC m=+0.424307071 container attach 8cde6114f5fe69484ea00b95cbea169ae04f38d9ac9249b4db8b47a97daf4159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0) 2024-08-19T01:53:30.166 INFO:journalctl@ceph.osd.10.smithi123.stdout:Aug 19 01:53:29 smithi123 podman[59901]: 2024-08-19 01:53:29.947654901 +0000 UTC m=+0.535999514 container died 8cde6114f5fe69484ea00b95cbea169ae04f38d9ac9249b4db8b47a97daf4159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:53:30.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:29.873+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:30.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:29.873+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:30.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:29.873+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:30.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:29.873+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:30.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:29.873+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:30.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:29.873+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:30.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:29 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:29.873+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:30.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.10.service' 2024-08-19T01:53:30.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:30.582 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-08-19T01:53:30.582 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-08-19T01:53:30.582 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.11 2024-08-19T01:53:30.890 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 systemd[1]: Stopping Ceph osd.11 for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:31.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.888+0000 7f2487c0f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:31.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.888+0000 7f2487c0f640 -1 osd.11 122 *** Got signal Terminated *** 2024-08-19T01:53:31.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.888+0000 7f2487c0f640 -1 osd.11 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T01:53:31.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.910+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:31.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.910+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:31.199 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.910+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:31.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.910+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:31.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.910+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:31.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.910+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:31.200 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:30 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:30.910+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:32.448 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:31.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:32.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:31.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:32.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:31.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:32.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:31.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:32.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:31.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:32.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:31.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:32.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:31 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:31.958+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:33.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:33.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:33.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:33.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:33.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:33.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:33.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:33.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.002+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6820 osd.8 since back 2024-08-19T01:53:06.553558+0000 front 2024-08-19T01:53:06.553591+0000 (oldest deadline 2024-08-19T01:53:32.954290+0000) 2024-08-19T01:53:34.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:34.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:34.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:34.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:34.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:34.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:34.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:34.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:33 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:33.967+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6820 osd.8 since back 2024-08-19T01:53:06.553558+0000 front 2024-08-19T01:53:06.553591+0000 (oldest deadline 2024-08-19T01:53:32.954290+0000) 2024-08-19T01:53:35.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-08-19T01:52:05.944568+0000 front 2024-08-19T01:52:05.944640+0000 (oldest deadline 2024-08-19T01:52:31.844677+0000) 2024-08-19T01:53:35.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-08-19T01:52:13.545694+0000 front 2024-08-19T01:52:13.545665+0000 (oldest deadline 2024-08-19T01:52:37.645888+0000) 2024-08-19T01:53:35.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-08-19T01:52:20.546775+0000 front 2024-08-19T01:52:20.547058+0000 (oldest deadline 2024-08-19T01:52:45.846990+0000) 2024-08-19T01:53:35.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-08-19T01:52:38.049522+0000 front 2024-08-19T01:52:38.049535+0000 (oldest deadline 2024-08-19T01:53:03.949417+0000) 2024-08-19T01:53:35.449 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-08-19T01:52:43.949781+0000 front 2024-08-19T01:52:43.949822+0000 (oldest deadline 2024-08-19T01:53:09.150348+0000) 2024-08-19T01:53:35.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-08-19T01:52:54.951890+0000 front 2024-08-19T01:52:54.951905+0000 (oldest deadline 2024-08-19T01:53:20.752278+0000) 2024-08-19T01:53:35.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-08-19T01:53:00.752945+0000 front 2024-08-19T01:53:00.752973+0000 (oldest deadline 2024-08-19T01:53:26.052744+0000) 2024-08-19T01:53:35.450 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:34 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11[46813]: 2024-08-19T01:53:34.948+0000 7f2484225640 -1 osd.11 122 heartbeat_check: no reply from 172.21.15.161:6820 osd.8 since back 2024-08-19T01:53:06.553558+0000 front 2024-08-19T01:53:06.553591+0000 (oldest deadline 2024-08-19T01:53:32.954290+0000) 2024-08-19T01:53:36.540 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:36 smithi161 podman[60884]: 2024-08-19 01:53:36.251361995 +0000 UTC m=+5.462907625 container died fced39acefc6dd2b7d014a02e5c593a9f7b717b4ca9d4878393f2fe1f9418b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-08-19T01:53:36.898 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:36 smithi161 podman[60884]: 2024-08-19 01:53:36.621696891 +0000 UTC m=+5.833242522 container cleanup fced39acefc6dd2b7d014a02e5c593a9f7b717b4ca9d4878393f2fe1f9418b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-19T01:53:36.898 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:36 smithi161 bash[60884]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11 2024-08-19T01:53:37.198 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:36 smithi161 podman[60898]: 2024-08-19 01:53:36.897834153 +0000 UTC m=+0.639284892 container remove fced39acefc6dd2b7d014a02e5c593a9f7b717b4ca9d4878393f2fe1f9418b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-08-19T01:53:37.517 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:37 smithi161 podman[60991]: 2024-08-19 01:53:37.124929056 +0000 UTC m=+0.017826368 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T01:53:37.518 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:37 smithi161 podman[60991]: 2024-08-19 01:53:37.231925035 +0000 UTC m=+0.124822341 container create 240d6a0466ed2f89ce4da351412a9f1b809bbff9a444c9cfe664870193aa8b55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-19T01:53:37.518 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:37 smithi161 podman[60991]: 2024-08-19 01:53:37.44234749 +0000 UTC m=+0.335244813 container init 240d6a0466ed2f89ce4da351412a9f1b809bbff9a444c9cfe664870193aa8b55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:53:37.518 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:37 smithi161 podman[60991]: 2024-08-19 01:53:37.44560344 +0000 UTC m=+0.338500749 container start 240d6a0466ed2f89ce4da351412a9f1b809bbff9a444c9cfe664870193aa8b55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-19T01:53:37.518 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:37 smithi161 podman[60991]: 2024-08-19 01:53:37.517370882 +0000 UTC m=+0.410268205 container attach 240d6a0466ed2f89ce4da351412a9f1b809bbff9a444c9cfe664870193aa8b55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:53:37.879 INFO:journalctl@ceph.osd.11.smithi161.stdout:Aug 19 01:53:37 smithi161 podman[60991]: 2024-08-19 01:53:37.63094376 +0000 UTC m=+0.523841074 container died 240d6a0466ed2f89ce4da351412a9f1b809bbff9a444c9cfe664870193aa8b55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-08-19T01:53:38.148 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@osd.11.service' 2024-08-19T01:53:38.185 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:38.186 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-08-19T01:53:38.186 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-08-19T01:53:38.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.a 2024-08-19T01:53:38.484 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:53:38 smithi064 systemd[1]: Stopping Ceph mds.a for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:38.984 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:53:38 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a[56609]: 2024-08-19T01:53:38.490+0000 7fe214fef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:38.984 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:53:38 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a[56609]: 2024-08-19T01:53:38.490+0000 7fe214fef640 -1 mds.a *** got signal Terminated *** 2024-08-19T01:53:42.808 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:53:42 smithi064 podman[87641]: 2024-08-19 01:53:42.496538245 +0000 UTC m=+4.108304200 container died dd537a28c24f2ae26e166b826c9090b35535f44a1a7cc048520ee6e33e5fc80e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-19T01:53:43.208 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:53:42 smithi064 podman[87641]: 2024-08-19 01:53:42.973650947 +0000 UTC m=+4.585416944 container cleanup dd537a28c24f2ae26e166b826c9090b35535f44a1a7cc048520ee6e33e5fc80e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-08-19T01:53:43.209 INFO:journalctl@ceph.mds.a.smithi064.stdout:Aug 19 01:53:42 smithi064 bash[87641]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-a 2024-08-19T01:53:43.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.a.service' 2024-08-19T01:53:43.379 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:43.380 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-08-19T01:53:43.380 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-08-19T01:53:43.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.d 2024-08-19T01:53:43.734 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:53:43 smithi064 systemd[1]: Stopping Ceph mds.d for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:43.734 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:53:43 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d[55386]: 2024-08-19T01:53:43.710+0000 7f3115d97640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:43.735 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:53:43 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d[55386]: 2024-08-19T01:53:43.710+0000 7f3115d97640 -1 mds.d *** got signal Terminated *** 2024-08-19T01:53:47.385 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:53:47 smithi064 podman[87761]: 2024-08-19 01:53:47.091548073 +0000 UTC m=+3.481047965 container died 41b5957e387d679190c72768a51bccc1c366f879245e01baa3c27288799b8adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-19T01:53:47.734 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:53:47 smithi064 podman[87761]: 2024-08-19 01:53:47.385753948 +0000 UTC m=+3.775253838 container cleanup 41b5957e387d679190c72768a51bccc1c366f879245e01baa3c27288799b8adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-08-19T01:53:47.734 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:53:47 smithi064 bash[87761]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d 2024-08-19T01:53:47.735 INFO:journalctl@ceph.mds.d.smithi064.stdout:Aug 19 01:53:47 smithi064 podman[87775]: 2024-08-19 01:53:47.60340374 +0000 UTC m=+0.507405427 container remove 41b5957e387d679190c72768a51bccc1c366f879245e01baa3c27288799b8adf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T01:53:47.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.d.service' 2024-08-19T01:53:47.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:47.797 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-08-19T01:53:47.797 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-08-19T01:53:47.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.g 2024-08-19T01:53:48.107 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:53:47 smithi064 systemd[1]: Stopping Ceph mds.g for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:48.484 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:53:48 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-g[56273]: 2024-08-19T01:53:48.106+0000 7f8c7ad22640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:48.484 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:53:48 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-g[56273]: 2024-08-19T01:53:48.106+0000 7f8c7ad22640 -1 mds.g *** got signal Terminated *** 2024-08-19T01:53:52.390 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:53:52 smithi064 podman[87881]: 2024-08-19 01:53:52.12025283 +0000 UTC m=+4.098851369 container died f5862aed040d9708f017cb23116a881afa85837aeab7eb7ce1bc7fb520c769b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-g, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-08-19T01:53:52.732 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:53:52 smithi064 podman[87881]: 2024-08-19 01:53:52.481148109 +0000 UTC m=+4.459746637 container cleanup f5862aed040d9708f017cb23116a881afa85837aeab7eb7ce1bc7fb520c769b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-g, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:53:52.732 INFO:journalctl@ceph.mds.g.smithi064.stdout:Aug 19 01:53:52 smithi064 bash[87881]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-g 2024-08-19T01:53:52.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.g.service' 2024-08-19T01:53:52.911 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:52.912 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-08-19T01:53:52.912 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-08-19T01:53:52.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.j 2024-08-19T01:53:53.243 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:53:52 smithi064 systemd[1]: Stopping Ceph mds.j for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:53.734 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:53:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j[54997]: 2024-08-19T01:53:53.242+0000 7f01d0a0b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:53.734 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:53:53 smithi064 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j[54997]: 2024-08-19T01:53:53.242+0000 7f01d0a0b640 -1 mds.j *** got signal Terminated *** 2024-08-19T01:53:55.984 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:53:55 smithi064 podman[88003]: 2024-08-19 01:53:55.714403602 +0000 UTC m=+2.582472446 container died 0b480a3d658ec620763bb4289c4124531ba4105994c71a9726eb524eeef4bf2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-19T01:53:56.447 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:53:56 smithi064 podman[88003]: 2024-08-19 01:53:56.026239673 +0000 UTC m=+2.894308516 container cleanup 0b480a3d658ec620763bb4289c4124531ba4105994c71a9726eb524eeef4bf2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-19T01:53:56.447 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:53:56 smithi064 bash[88003]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j 2024-08-19T01:53:56.447 INFO:journalctl@ceph.mds.j.smithi064.stdout:Aug 19 01:53:56 smithi064 podman[88029]: 2024-08-19 01:53:56.218967668 +0000 UTC m=+0.506800953 container remove 0b480a3d658ec620763bb4289c4124531ba4105994c71a9726eb524eeef4bf2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-j, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-08-19T01:53:56.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.j.service' 2024-08-19T01:53:56.496 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:53:56.496 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-08-19T01:53:56.496 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-08-19T01:53:56.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.b 2024-08-19T01:53:56.899 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:53:56 smithi123 systemd[1]: Stopping Ceph mds.b for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:53:57.165 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:53:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: 2024-08-19T01:53:56.896+0000 7f01402df640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:53:57.166 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:53:56 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b[51735]: 2024-08-19T01:53:56.896+0000 7f01402df640 -1 mds.b *** got signal Terminated *** 2024-08-19T01:53:59.868 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:53:59 smithi123 podman[60003]: 2024-08-19 01:53:59.483281849 +0000 UTC m=+2.786526360 container died 532c3f8b51c467033319bf770c7810973cd4fc3a86c34aae7823f2486bd7b60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b, ceph=True, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:54:00.165 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:53:59 smithi123 podman[60003]: 2024-08-19 01:53:59.972707912 +0000 UTC m=+3.275952426 container cleanup 532c3f8b51c467033319bf770c7810973cd4fc3a86c34aae7823f2486bd7b60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-19T01:54:00.165 INFO:journalctl@ceph.mds.b.smithi123.stdout:Aug 19 01:53:59 smithi123 bash[60003]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-b 2024-08-19T01:54:00.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.b.service' 2024-08-19T01:54:00.442 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:00.443 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-08-19T01:54:00.443 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-08-19T01:54:00.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.e 2024-08-19T01:54:00.665 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:54:00 smithi123 systemd[1]: Stopping Ceph mds.e for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:54:01.165 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:54:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e[51127]: 2024-08-19T01:54:00.759+0000 7f679a608640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:54:01.165 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:54:00 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e[51127]: 2024-08-19T01:54:00.759+0000 7f679a608640 -1 mds.e *** got signal Terminated *** 2024-08-19T01:54:03.763 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:54:03 smithi123 podman[60125]: 2024-08-19 01:54:03.485269632 +0000 UTC m=+2.823289647 container died afb9d2fa33f82c612be9ac103e10cf4962a795d6eb799b4f4e908b69271647e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:54:04.119 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:54:03 smithi123 podman[60125]: 2024-08-19 01:54:03.859804693 +0000 UTC m=+3.197824707 container cleanup afb9d2fa33f82c612be9ac103e10cf4962a795d6eb799b4f4e908b69271647e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-08-19T01:54:04.120 INFO:journalctl@ceph.mds.e.smithi123.stdout:Aug 19 01:54:03 smithi123 bash[60125]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-e 2024-08-19T01:54:04.257 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.e.service' 2024-08-19T01:54:04.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:04.292 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-08-19T01:54:04.292 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-08-19T01:54:04.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.h 2024-08-19T01:54:04.631 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:54:04 smithi123 systemd[1]: Stopping Ceph mds.h for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:54:04.915 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:54:04 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h[51430]: 2024-08-19T01:54:04.630+0000 7f1d78fbc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:54:04.915 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:54:04 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h[51430]: 2024-08-19T01:54:04.630+0000 7f1d78fbc640 -1 mds.h *** got signal Terminated *** 2024-08-19T01:54:07.748 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:54:07 smithi123 podman[60260]: 2024-08-19 01:54:07.472246653 +0000 UTC m=+2.948285754 container died 7b76ed2d44d7765b33fbf8c295109e0eed1e738d3a84fc1b34c3cb760a2e5026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-08-19T01:54:08.106 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:54:07 smithi123 podman[60260]: 2024-08-19 01:54:07.830282435 +0000 UTC m=+3.306321536 container cleanup 7b76ed2d44d7765b33fbf8c295109e0eed1e738d3a84fc1b34c3cb760a2e5026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-19T01:54:08.107 INFO:journalctl@ceph.mds.h.smithi123.stdout:Aug 19 01:54:07 smithi123 bash[60260]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-h 2024-08-19T01:54:08.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.h.service' 2024-08-19T01:54:08.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:08.307 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-08-19T01:54:08.307 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-08-19T01:54:08.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.k 2024-08-19T01:54:08.642 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:54:08 smithi123 systemd[1]: Stopping Ceph mds.k for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:54:08.915 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:54:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k[50808]: 2024-08-19T01:54:08.641+0000 7f26e3f42640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:54:08.915 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:54:08 smithi123 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k[50808]: 2024-08-19T01:54:08.641+0000 7f26e3f42640 -1 mds.k *** got signal Terminated *** 2024-08-19T01:54:12.915 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:54:12 smithi123 podman[60381]: 2024-08-19 01:54:12.635727307 +0000 UTC m=+4.101326452 container died 0bd66baa8cf233531bc3d9484eba788f7844223fc134167d39bc97fe8931bccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-19T01:54:13.276 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:54:13 smithi123 podman[60381]: 2024-08-19 01:54:13.017061086 +0000 UTC m=+4.482660237 container cleanup 0bd66baa8cf233531bc3d9484eba788f7844223fc134167d39bc97fe8931bccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-08-19T01:54:13.277 INFO:journalctl@ceph.mds.k.smithi123.stdout:Aug 19 01:54:13 smithi123 bash[60381]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-k 2024-08-19T01:54:13.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.k.service' 2024-08-19T01:54:13.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:13.428 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-08-19T01:54:13.428 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-08-19T01:54:13.429 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.c 2024-08-19T01:54:13.837 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:13 smithi161 systemd[1]: Stopping Ceph mds.c for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:54:14.199 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c[51957]: 2024-08-19T01:54:13.836+0000 7f03bd18d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:54:14.199 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:13 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c[51957]: 2024-08-19T01:54:13.836+0000 7f03bd18d640 -1 mds.c *** got signal Terminated *** 2024-08-19T01:54:15.938 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:15 smithi161 podman[61094]: 2024-08-19 01:54:15.477444822 +0000 UTC m=+1.836441472 container died 1e98d414f9a496b881f72bda6fe80c9292dcd34c33ff4ba73e8c80f6f1df37e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-19T01:54:16.188 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:16 smithi161 podman[61094]: 2024-08-19 01:54:16.019799674 +0000 UTC m=+2.378796323 container cleanup 1e98d414f9a496b881f72bda6fe80c9292dcd34c33ff4ba73e8c80f6f1df37e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T01:54:16.188 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:16 smithi161 bash[61094]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c 2024-08-19T01:54:16.432 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.c.service' 2024-08-19T01:54:16.448 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:16 smithi161 podman[61117]: 2024-08-19 01:54:16.187944772 +0000 UTC m=+0.705301674 container remove 1e98d414f9a496b881f72bda6fe80c9292dcd34c33ff4ba73e8c80f6f1df37e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-c, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T01:54:16.448 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:16 smithi161 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.c.service: Deactivated successfully. 2024-08-19T01:54:16.448 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:16 smithi161 systemd[1]: Stopped Ceph mds.c for 88c1a048-5dc9-11ef-bccf-c7b262605968. 2024-08-19T01:54:16.449 INFO:journalctl@ceph.mds.c.smithi161.stdout:Aug 19 01:54:16 smithi161 systemd[1]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.c.service: Consumed 10.017s CPU time. 2024-08-19T01:54:16.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:16.469 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-08-19T01:54:16.469 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-08-19T01:54:16.470 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.f 2024-08-19T01:54:16.817 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:54:16 smithi161 systemd[1]: Stopping Ceph mds.f for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:54:17.198 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:54:16 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f[51646]: 2024-08-19T01:54:16.815+0000 7fa10c464640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:54:17.199 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:54:16 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f[51646]: 2024-08-19T01:54:16.815+0000 7fa10c464640 -1 mds.f *** got signal Terminated *** 2024-08-19T01:54:19.733 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:54:19 smithi161 podman[61225]: 2024-08-19 01:54:19.471198528 +0000 UTC m=+2.774441402 container died 3a9cfe8416775d08a129e94658cb10da919b1c72476d981d73bacedb5a72eb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-08-19T01:54:20.100 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:54:19 smithi161 podman[61225]: 2024-08-19 01:54:19.832305989 +0000 UTC m=+3.135548872 container cleanup 3a9cfe8416775d08a129e94658cb10da919b1c72476d981d73bacedb5a72eb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-19T01:54:20.101 INFO:journalctl@ceph.mds.f.smithi161.stdout:Aug 19 01:54:19 smithi161 bash[61225]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-f 2024-08-19T01:54:20.289 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.f.service' 2024-08-19T01:54:20.322 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:20.323 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-08-19T01:54:20.323 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-08-19T01:54:20.323 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.i 2024-08-19T01:54:20.698 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:54:20 smithi161 systemd[1]: Stopping Ceph mds.i for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:54:20.699 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:54:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i[52261]: 2024-08-19T01:54:20.627+0000 7fd0d60e0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:54:20.699 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:54:20 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i[52261]: 2024-08-19T01:54:20.627+0000 7fd0d60e0640 -1 mds.i *** got signal Terminated *** 2024-08-19T01:54:23.811 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:54:23 smithi161 podman[61346]: 2024-08-19 01:54:23.47671821 +0000 UTC m=+2.933062856 container died 8a581b31383a95413781cb77fd1ea3cb78a53bbf310016e09a3b6c11d44f738e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=squid-de09dd0) 2024-08-19T01:54:24.198 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:54:23 smithi161 podman[61346]: 2024-08-19 01:54:23.910824993 +0000 UTC m=+3.367169640 container cleanup 8a581b31383a95413781cb77fd1ea3cb78a53bbf310016e09a3b6c11d44f738e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-08-19T01:54:24.199 INFO:journalctl@ceph.mds.i.smithi161.stdout:Aug 19 01:54:23 smithi161 bash[61346]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-i 2024-08-19T01:54:24.354 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.i.service' 2024-08-19T01:54:24.386 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:24.387 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-08-19T01:54:24.387 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-08-19T01:54:24.387 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.l 2024-08-19T01:54:24.757 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:54:24 smithi161 systemd[1]: Stopping Ceph mds.l for 88c1a048-5dc9-11ef-bccf-c7b262605968... 2024-08-19T01:54:25.198 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:54:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l[51328]: 2024-08-19T01:54:24.756+0000 7fafa4b42640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T01:54:25.199 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:54:24 smithi161 ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l[51328]: 2024-08-19T01:54:24.756+0000 7fafa4b42640 -1 mds.l *** got signal Terminated *** 2024-08-19T01:54:26.198 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:54:25 smithi161 podman[61467]: 2024-08-19 01:54:25.942813869 +0000 UTC m=+1.335539144 container died 3427f5f088bb8d98c5d5e7697ffeff70b84efdbebc750c8a27d58ea64a04234a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-08-19T01:54:26.599 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:54:26 smithi161 podman[61467]: 2024-08-19 01:54:26.289924848 +0000 UTC m=+1.682650112 container cleanup 3427f5f088bb8d98c5d5e7697ffeff70b84efdbebc750c8a27d58ea64a04234a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-19T01:54:26.599 INFO:journalctl@ceph.mds.l.smithi161.stdout:Aug 19 01:54:26 smithi161 bash[61467]: ceph-88c1a048-5dc9-11ef-bccf-c7b262605968-mds-l 2024-08-19T01:54:26.729 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-88c1a048-5dc9-11ef-bccf-c7b262605968@mds.l.service' 2024-08-19T01:54:26.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T01:54:26.761 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-08-19T01:54:26.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-19T01:54:26.972 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:54:29.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-19T01:54:29.615 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:54:32.100 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-19T01:54:32.304 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T01:54:34.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:54:34.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:54:34.925 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T01:54:34.952 INFO:tasks.cephadm:Archiving crash dumps... 2024-08-19T01:54:34.954 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi064/crash 2024-08-19T01:54:34.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash -- . 2024-08-19T01:54:34.994 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-19T01:54:34.994 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-08-19T01:54:34.996 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi123/crash 2024-08-19T01:54:34.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash -- . 2024-08-19T01:54:35.036 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-19T01:54:35.036 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-08-19T01:54:35.038 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi161/crash 2024-08-19T01:54:35.042 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash -- . 2024-08-19T01:54:35.083 INFO:teuthology.orchestra.run.smithi161.stderr:tar: /var/lib/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-19T01:54:35.083 INFO:teuthology.orchestra.run.smithi161.stderr:tar: Error is not recoverable: exiting now 2024-08-19T01:54:35.084 INFO:tasks.cephadm:Checking cluster log for badness... 2024-08-19T01:54:35.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-08-19T01:54:35.123 INFO:tasks.cephadm:Compressing logs... 2024-08-19T01:54:35.124 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T01:54:35.166 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T01:54:35.169 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T01:54:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-19T01:54:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2024-08-19T01:54:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-08-19T01:54:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-19T01:54:35.192 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.3.log 2024-08-19T01:54:35.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.log: /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.d.log 2024-08-19T01:54:35.192 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.x.log 2024-08-19T01:54:35.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.3.log: /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-19T01:54:35.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-19T01:54:35.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-08-19T01:54:35.194 INFO:teuthology.orchestra.run.smithi064.stderr: 84.8% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-19T01:54:35.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.a.log 2024-08-19T01:54:35.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.a.log 2024-08-19T01:54:35.196 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.a.log: 90.3% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-19T01:54:35.198 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.9.log 2024-08-19T01:54:35.198 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.d.log.gz 2024-08-19T01:54:35.199 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.6.log 2024-08-19T01:54:35.200 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.9.log: 94.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-19T01:54:35.200 INFO:teuthology.orchestra.run.smithi064.stderr: 90.9% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-19T01:54:35.203 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.a.log.gz 2024-08-19T01:54:35.204 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.j.log 2024-08-19T01:54:35.204 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.0.log 2024-08-19T01:54:35.206 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log 2024-08-19T01:54:35.206 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.g.log 2024-08-19T01:54:35.208 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-08-19T01:54:35.208 INFO:teuthology.orchestra.run.smithi123.stderr: /var/log/ceph/cephadm.log 2024-08-19T01:54:35.208 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-19T01:54:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.e.log 2024-08-19T01:54:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.h.log 2024-08-19T01:54:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-19T01:54:35.210 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log: 87.1% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-19T01:54:35.212 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.b.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.10.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.4.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.k.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.b.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.y.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.7.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.1.log 2024-08-19T01:54:35.213 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log 2024-08-19T01:54:35.214 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.g.log: 94.6% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.j.log.gz 2024-08-19T01:54:35.216 INFO:teuthology.orchestra.run.smithi064.stderr: 94.7% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.g.log.gz 2024-08-19T01:54:35.221 INFO:teuthology.orchestra.run.smithi161.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-19T01:54:35.221 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-19T01:54:35.221 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-19T01:54:35.223 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.z.log 2024-08-19T01:54:35.223 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.2.log 2024-08-19T01:54:35.223 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.11.log 2024-08-19T01:54:35.223 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.f.log 2024-08-19T01:54:35.225 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-19T01:54:35.225 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-19T01:54:35.225 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.8.log 2024-08-19T01:54:35.226 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log: -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.i.log 2024-08-19T01:54:35.226 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.c.log 2024-08-19T01:54:35.226 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.5.log 2024-08-19T01:54:35.226 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log 2024-08-19T01:54:35.226 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.c.log: /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.c.log 2024-08-19T01:54:35.226 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.l.log 2024-08-19T01:54:35.229 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.c.log: /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.l.log.gz 2024-08-19T01:54:35.353 INFO:teuthology.orchestra.run.smithi161.stderr: 82.2% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-19T01:54:35.358 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.1.log: /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-19T01:54:35.358 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-19T01:54:35.372 INFO:teuthology.orchestra.run.smithi161.stderr: 92.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.z.log.gz 2024-08-19T01:54:35.374 INFO:teuthology.orchestra.run.smithi161.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-19T01:54:35.375 INFO:teuthology.orchestra.run.smithi123.stderr: 82.2% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-19T01:54:35.407 INFO:teuthology.orchestra.run.smithi123.stderr: 90.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-19T01:54:35.408 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-19T01:54:35.429 INFO:teuthology.orchestra.run.smithi123.stderr: 94.5% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.k.log.gz 2024-08-19T01:54:35.491 INFO:teuthology.orchestra.run.smithi123.stderr: 92.3% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.y.log.gz 2024-08-19T01:54:35.527 INFO:teuthology.orchestra.run.smithi064.stderr: 90.3% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mgr.x.log.gz 2024-08-19T01:54:35.528 INFO:teuthology.orchestra.run.smithi161.stderr: 87.1% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-19T01:54:35.884 INFO:teuthology.orchestra.run.smithi123.stderr: 90.9% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-19T01:54:36.459 INFO:teuthology.orchestra.run.smithi161.stderr: 90.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-19T01:54:36.926 INFO:teuthology.orchestra.run.smithi123.stderr: 91.7% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.b.log.gz 2024-08-19T01:54:37.107 INFO:teuthology.orchestra.run.smithi064.stderr: 91.9% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.a.log.gz 2024-08-19T01:54:40.741 INFO:teuthology.orchestra.run.smithi161.stderr: 95.2% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.c.log.gz 2024-08-19T01:54:42.221 INFO:teuthology.orchestra.run.smithi161.stderr: 91.9% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mon.c.log.gz 2024-08-19T01:54:49.635 INFO:teuthology.orchestra.run.smithi123.stderr: 94.3% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.e.log.gz 2024-08-19T01:55:01.689 INFO:teuthology.orchestra.run.smithi064.stderr: 91.7% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.0.log.gz 2024-08-19T01:55:07.776 INFO:teuthology.orchestra.run.smithi064.stderr: 91.2% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.6.log.gz 2024-08-19T01:55:08.928 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.3.log.gz 2024-08-19T01:55:10.890 INFO:teuthology.orchestra.run.smithi064.stderr: 91.3% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.9.log.gz 2024-08-19T01:55:10.892 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-08-19T01:55:10.892 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m35.713s 2024-08-19T01:55:10.893 INFO:teuthology.orchestra.run.smithi064.stderr:user 2m6.796s 2024-08-19T01:55:10.893 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m3.742s 2024-08-19T01:55:14.477 INFO:teuthology.orchestra.run.smithi123.stderr: 91.5% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.10.log.gz 2024-08-19T01:55:45.444 INFO:teuthology.orchestra.run.smithi123.stderr: 91.7% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.1.log.gz 2024-08-19T01:55:59.945 INFO:teuthology.orchestra.run.smithi123.stderr: 91.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.4.log.gz 2024-08-19T01:56:01.373 INFO:teuthology.orchestra.run.smithi123.stderr: 91.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.7.log.gz 2024-08-19T01:56:13.779 INFO:teuthology.orchestra.run.smithi161.stderr: 95.5% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.f.log.gz 2024-08-19T01:56:29.146 INFO:teuthology.orchestra.run.smithi123.stderr: 93.6% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.h.log.gz 2024-08-19T01:57:06.419 INFO:teuthology.orchestra.run.smithi123.stderr: 93.3% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.b.log.gz 2024-08-19T01:57:06.421 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-08-19T01:57:06.421 INFO:teuthology.orchestra.run.smithi123.stderr:real 2m31.238s 2024-08-19T01:57:06.421 INFO:teuthology.orchestra.run.smithi123.stderr:user 4m22.911s 2024-08-19T01:57:06.422 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m9.324s 2024-08-19T01:57:36.978 INFO:teuthology.orchestra.run.smithi161.stderr: 91.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.2.log.gz 2024-08-19T01:58:11.520 INFO:teuthology.orchestra.run.smithi161.stderr: 91.4% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.11.log.gz 2024-08-19T01:59:32.810 INFO:teuthology.orchestra.run.smithi161.stderr: 91.3% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.8.log.gz 2024-08-19T02:00:26.595 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-osd.5.log.gz 2024-08-19T02:04:29.276 INFO:teuthology.orchestra.run.smithi161.stderr: 93.5% -- replaced with /var/log/ceph/88c1a048-5dc9-11ef-bccf-c7b262605968/ceph-mds.i.log.gz 2024-08-19T02:04:29.278 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-08-19T02:04:29.278 INFO:teuthology.orchestra.run.smithi161.stderr:real 9m54.095s 2024-08-19T02:04:29.278 INFO:teuthology.orchestra.run.smithi161.stderr:user 6m18.855s 2024-08-19T02:04:29.278 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m16.694s 2024-08-19T02:04:29.279 INFO:tasks.cephadm:Archiving logs... 2024-08-19T02:04:29.279 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi064/log 2024-08-19T02:04:29.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T02:04:50.082 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi123/log 2024-08-19T02:04:50.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T02:05:30.772 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi161/log 2024-08-19T02:05:30.774 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T02:06:30.734 INFO:tasks.cephadm:Removing cluster... 2024-08-19T02:06:30.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 --force 2024-08-19T02:06:30.946 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T02:06:31.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 --force 2024-08-19T02:06:31.606 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T02:06:32.182 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 88c1a048-5dc9-11ef-bccf-c7b262605968 --force 2024-08-19T02:06:32.720 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: 88c1a048-5dc9-11ef-bccf-c7b262605968 2024-08-19T02:06:33.364 INFO:tasks.cephadm:Removing cephadm ... 2024-08-19T02:06:33.365 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-19T02:06:33.384 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-19T02:06:33.552 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-19T02:06:33.574 INFO:tasks.cephadm:Teardown complete 2024-08-19T02:06:33.575 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-19T02:06:33.632 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-19T02:06:33.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T02:06:33.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T02:06:33.659 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T02:06:33.776 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-19T02:06:33.998 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-19T02:06:33.998 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T02:06:34.002 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T02:06:34.005 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T02:06:34.017 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-08-19T02:06:34.020 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-08-19T02:06:34.020 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-08-19T02:06:34.022 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T02:06:34.022 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-08-19T02:06:34.022 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +313us[ +278us] +/- 43ms 2024-08-19T02:06:34.023 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 -287us[ -322us] +/- 49ms 2024-08-19T02:06:34.023 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -144us[ -179us] +/- 69ms 2024-08-19T02:06:34.023 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T02:06:34.052 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T02:06:34.052 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-08-19T02:06:34.053 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +326us[ +288us] +/- 44ms 2024-08-19T02:06:34.053 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 -284us[ -326us] +/- 49ms 2024-08-19T02:06:34.053 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -119us[ -119us] +/- 70ms 2024-08-19T02:06:34.053 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T02:06:34.063 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T02:06:34.063 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-08-19T02:06:34.063 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +237us[ +229us] +/- 43ms 2024-08-19T02:06:34.063 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -384us[ -384us] +/- 49ms 2024-08-19T02:06:34.063 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -220us[ -220us] +/- 70ms 2024-08-19T02:06:34.064 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T02:06:34.064 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-19T02:06:34.343 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-19T02:06:34.344 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-19T02:06:34.372 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T02:06:34.446 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T02:06:34.531 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T02:06:34.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T02:06:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T02:06:34.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T02:06:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T02:06:34.672 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T02:06:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T02:06:34.700 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-19T02:06:34.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-19T02:06:34.722 INFO:teuthology.task.internal:Duration was 2846.762057 seconds 2024-08-19T02:06:34.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-19T02:06:34.733 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-19T02:06:34.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T02:06:34.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T02:06:34.738 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T02:06:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T02:06:34.804 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T02:06:34.806 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T02:06:35.275 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-19T02:06:35.275 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-08-19T02:06:35.275 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T02:06:35.338 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-08-19T02:06:35.338 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T02:06:35.371 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-08-19T02:06:35.371 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T02:06:35.395 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-19T02:06:35.396 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T02:06:35.398 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T02:06:35.414 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T02:06:35.471 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-19T02:06:35.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T02:06:35.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T02:06:35.483 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T02:06:36.237 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-19T02:06:36.249 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-19T02:06:36.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T02:06:36.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T02:06:36.312 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T02:06:36.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-19T02:06:36.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T02:06:36.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T02:06:36.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T02:06:36.388 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-08-19T02:06:36.391 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-08-19T02:06:36.432 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-08-19T02:06:36.455 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T02:06:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T02:06:36.470 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T02:06:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T02:06:36.485 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T02:06:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T02:06:36.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-19T02:06:36.523 INFO:teuthology.task.internal:Transferring archived files... 2024-08-19T02:06:36.525 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi064 2024-08-19T02:06:36.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T02:06:36.578 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi123 2024-08-19T02:06:36.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T02:06:36.612 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_21:24:02-fs-squid-distro-default-smithi/7860119/remote/smithi161 2024-08-19T02:06:36.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T02:06:36.646 INFO:teuthology.task.internal:Removing archive directory... 2024-08-19T02:06:36.646 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T02:06:36.649 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T02:06:36.651 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T02:06:36.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-19T02:06:36.710 INFO:teuthology.task.internal:Not uploading archives. 2024-08-19T02:06:36.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-19T02:06:36.721 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-19T02:06:36.722 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T02:06:36.724 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T02:06:36.726 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T02:06:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 02:06 /home/ubuntu/cephtest 2024-08-19T02:06:36.738 INFO:teuthology.orchestra.run.smithi123.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 02:06 /home/ubuntu/cephtest 2024-08-19T02:06:36.751 INFO:teuthology.orchestra.run.smithi161.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 02:06 /home/ubuntu/cephtest 2024-08-19T02:06:36.753 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-19T02:06:36.764 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-19T02:06:36.794 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 2846.7620573043823 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi064 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=78215f5a1cf5eaf321d506a66539942e12851f34 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e821b3b17ae146a7ac5cba89cd8065ea status: fail success: false 2024-08-19T02:06:36.794 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T02:06:36.875 INFO:teuthology.run:FAIL